Microsoft has blamed “operator error” for the multi-hour outage of its cloud SQL Server in Europe last week.
“Between 03:47 UTC and 13:30 UTC on 21 July 2022, customers using SQL Database and SQL Data Warehouse in West Europe may have experienced issues accessing services,” said Microsoft.
The issues were severe for affected customers. Attempting to make new connections to databases in the West Europe region resulted in errors and time-outs. While existing connections were OK, if they were closed, attempts to re-establish them faced the same issues.
And, of course, when Microsoft’s SQL Database gets ill, so too do many of the services that depend upon it, including App Services, Automation, Backup and so on.
It took almost three hours to achieve partial recovery (at 06:12 UTC) and the company said the problem was solved at 13:30 UTC, although it didn’t declare full mitigation until 18:45 UTC (“No failures that occurred after 13:30 UTC were directly as a result of this incident,” Microsoft said.)
So what happened? A PICNIC (Problem In Chair Not In Computer) by the sounds of things – “an operator error led to an incorrect action being performed in close sequence on all four persisted metadata caches,” explained Microsoft.
Connections to the Azure SQL Database service are dealt with by regional gateway clusters (West Europe has two) and there are multiple persisted metadata caches used for connection routing (again, West Europe has two per gateway.)
That “operator error” meant the caches were unavailable to the gateways. Gateway processes in the West Europe region couldn’t access connection routing metadata, and the incident kicked off.
Once the error was identified, engineers were faced with a decision to either revive the caches or rebuild entirely new ones. With the latter choice likely to take a lot longer than the former, engineers got cracking with fixing what was already in place.
By 06:18 UTC, success rates hit approximately 60 per cent, but issues persisted. “Firstly,” said Microsoft, “a timing issue in applying mitigation caused gateways in one of the two clusters to cache incorrect cache connection strings. Secondly, the metadata caches were not receiving updates for changes that happened while the caches were unavailable.”
Cue a careful restart of all the gateway nodes in the cluster and a script to deal with stale cache entries (where updates had been missed.)
Missing from Microsoft’s detailed explanation for the outage was the fate of the unfortunate operator whose Who, Me? moment caused such chaos for customers in the West Europe region. It was not explained how one person could wreak such havoc. Perhaps that PICNIC had a side of iffy processes.
Instead, the company closed the stable door long after the horse bolted by “programmatically blocking any further executions of the action that led to the metadata caches becoming unavailable.”
It has also thrown up stronger guardrails “to prevent human errors like the one that triggered the start of this incident.”
In-memory caching of connection routing metadata is also to be implemented and the company is to take a long, hard look at service resiliency. ®
Emperor Penguin Linus Torvalds has released the first release candidate for Linux 6.0, but doesn’t mind what you call it.
“After I had already decided to call this kernel 6.0, a few Chinese developers piped up and pointed out that ‘5.20’ is a more wholesome version of the Western ‘4.20’ internet-famous number,” he wrote in his announcement that Linux 6.0 rc1 has been released.
“4.20” is a reference to a day on which some celebrate marijuana, while “5.20” does likewise for magic mushrooms.
“So if you want to call this ‘Linux 5.20’, go right ahead,” Torvalds wrote.
“Because the kernel version numbers really are entirely made up and have no intrinsic meaning.”
That this week’s release has the 6.0 label is still nice to know, as discussion on the Linux kernel mailing list in recent weeks used 5.20 and 6.0 interchangeably.
As The Register has already reported, the release does not make major changes to the kernel but does include many useful updates – such as more RISC-V support, code to drive Intel’s Gaudi accelerators, and improved ACPI handling.
Torvalds lamented some Rust-enabling code didn’t make it into the release.
“I actually was hoping that we’d get some of the first rust infrastructure, and the multi-gen LRU VM, but neither of them happened this time around,” he mused, before observing “There’s always more releases.”
“This is one of those releases where you should not look at the diffstat too closely, because more than half of it is yet another AMD GPU register dump,” he added, noting that Intel’s Gaudi2 Ai processors are also likely to produce plenty of similar kernel additions.
“The CPU people also show up in the JSON files that describe the perf events, but they look absolutely tiny compared to the ‘asic_reg’ auto-generated GPU and AI hardware definitions,” he added.
The release includes 13,099 changed files, 1,280,295 insertions and 341,210 deletions. Torvalds calculated those numbers “just because I was curious and looked.”
He wants you to be curious too – or at least curious enough to test the kernel, because that’s what release candidates are for and this one contains at least one active bug. ®
Ireland is one of 19 countries worldwide that strongly dislikes Tinder. One in five Tweets by Irish people about all apps are negative.
According to Electronics Hub’s analysis of the most hated apps in the world, Tinder is the most loathed app in Ireland.
Irish people are not alone in their hatred for the dating app. Tinder was the most hated app in 19 countries in total, with Canadians, Americans, Nigerians, Kenyans and our neighbours in the UK also singling it out as their least favourite.
Electronics Hub determined the most hated apps in each country by analysing Twitter data. It processed more than 3m geotagged tweets related to 87 social media, dating, mobile games, entertainment, cryptocurrency and money transfer apps.
Researchers calculated the percentage of tweets about each app that were negative using a sentiment analysis tool which identifies whether a tweet has positive, negative or neutral sentiment.
Click to enlarge and see the most hated apps in the world by country. Infographic: Electronics Hub
Ireland was found to be one of the most negative countries when it came to attitudes towards apps. One in five Tweets posted by Irish people about apps were negative, Electronics Hub found.
Despite Irish people’s professed loathing for Tinder, the dating platform tried to play a role in keeping daters safe in the pandemic. It hooked up with the HSE to promote vaccines by adding badges to users’ profiles.
Tinder was only the second-most hated app in the world, with Roblox taking first place. More than 20 countries said the child-targeted gaming app was their most hated app. Other unpopular apps include Snapchat, Disney and Reddit.
Neighbouring countries tend to dislike similar apps, with the Scandinavians professing a dislike for Reddit and South Americans hating e-commerce apps.
Dating apps, meanwhile, are disliked the world over. In Iraq, 71.4pc of all tweets about Tinder are negative, which is the highest out of any country. A state-by-state breakdown of the most hated apps in North America also found Tinder took the top spot in 21 states.
10 things you need to know direct to your inbox every weekday. Sign up for the Daily Brief, Silicon Republic’s digest of essential sci-tech news.
Amazon workers say they are working in a “sweatshop” as safety concerns and worries about the cost of living crisis have triggered walkouts at warehouses around the country.
The Observer has spoken to four staff involved in the walkouts, who work at three Amazon warehouses, including Tilbury in Essex, where protests began on 4 August. All say they will struggle to survive this winter with pay rise offers between 35p and 50p an hour – far less than the rate of inflation, which is currently at 9.4%.
The workers, who spoke anonymously for fear of reprisals from Amazon, said they were speaking out to highlight how the firm’s ultra-cheap, ultra-convenient, super-fast delivery model works.
Amazon employs more than 70,000 people in the UK, adding 25,000 staff in 2021 alone. Many work at the company’s 21 fulfilment centres, where some workers say they are asked to carry out long, physical shifts, with difficult targets, for low pay.
Starting pay in Amazon warehouses will shortly be increasing to between £10.50 and £11.45 per hour, depending on location. An Amazon spokesperson saidthis was a 29% increase in the minimum hourly wage paid to staff since 2018. They said it is also augmented by a comprehensive benefits package worth thousands of pounds a year, and a company pension plan.
But staff say it is too low for the type of work being done and given the current economic crisis, especially at a company that just posted $121bn (£100bn) in revenues in the second quarter of 2022 alone.
“When we heard the news, it was shocking,” said one worker at Amazon’s warehouse in Tilbury. “It’s ridiculous. Inflation is [forecast to reach] 13%, and our salary increases barely 3%.” The worker rents a house with her husband for £1,350 a month without bills. “My salary is £1,600. … I’m lucky I’m married, otherwise I’d be homeless.”
Some staff are seeking a pay rise of £2 an hour from the tech giant.
Another worker at Amazon’s warehouse in Tilbury said they were “petrified” about how they would survive this winter. “We had a scenario recently where someone was living in [an] Amazon [warehouse],” he said. “If I’m honest, I can probably see that happening again.
“I can see people staying in the canteen all the time because they can’t afford to go home.”
The worker is protesting against the poor pay offer, as well as conditions that lock staff in cages for entire shifts at the warehouses, from where they pick items to be delivered to customers. (Amazon says the workstations are to protect workers from moving robotics.)
“It’s a Chinese sweatshop in the UK,” said the second worker at Tilbury. “It’s how they set up their model.”
The worker has struggled with his mental health while working for the company. “I’ve realised how bad Amazon is for my mental health,” he said. “The anxiety of going into work, knowing you’ve got to do the same stuff day in, day out, is horrible.”
That concern is echoed by a worker at an Amazon facility near Bristol, who has worked there with his wife for three years. “It was good initially,” the worker said. “There was a lot of safety consciousness, and the targets were pretty reasonable. But now they’re just pushing it higher and higher, and exploiting people.”
Around 100 Amazon staff at Bristol staged a sit-in at the company canteen on 10 August – action for which they say they were docked pay by management at the site. “The vast majority of people went back to work at that point, because at the end of the day, as much as they want to fight for it, they have to think about themselves financially.”
The Bristol warehouse worker says that managers used to stop employees from lifting heavy items from bins on high shelves in the warehouse without a ladder. “If you overstretched yourself for 10 hours, you’d end up with a bad neck and a bad back,” he said.
That has subsequently changed as staff said they felt pressured to meet ever-escalating demand. Staff pushing carts around the warehouse used to be limited to using one cart at a time for safety reasons; now it is claimed managers turn a blind eye to staff pulling two carts at once. “They don’t say nothing because all they care about is getting the work done as fast as possible,” he said. “Safety just goes out the window.”
He says he has personally lifted items weighing up to 25kg by himself, despite rules saying anything heavier than 15kg should be lifted by two people.
A worker at an Amazon facility in the north-west of Englandsaid that managers at his warehouse similarly ignored rules around not running on site and lifting down heavy items from high areas in an attempt to meet targets, which at his site require two items to be picked every minute.
Amazon declined to respond to specific claims.
Martha Dark, director at Foxglove, a non-profit organisation working to highlight issues within tech companies that supports Amazon workers, said: “None of the workers we’re supporting wanted to protest.
“They’re desperate and can’t survive on these wages. Meanwhile, Amazon threatens to dock pay and send workers to HR for revealing the truth about life in the warehouse.”
She added: “Amazon needs to respect workers’ rights to organise, stop penalising people who are fighting to survive and provide a real pay rise now.”
Two workers said they plan to leave the company because of the conditions and pay. However, some hope to stay put – to change things.
“If a lot of us who are experienced leave Amazon at this point they’ll get a new group of people in who they can mould into this depressing way of work,” said the Bristol worker. “That’s the problem.”