Meta will release a giant language model to academics, in hope that better understanding of how these systems work can make them less toxic and biased.
The Open Pretrained Transformer (OPT-175B) has 175 billion parameters, matching commercial language models like OpenAI’s GPT-3. These types of systems have introduced capabilities for developers to build upon like automated copywriting, content moderation, or even coding. But they can generate text that’s biased, toxic, and inaccurate, making them risky to use.
As Meta knows only too well from some of the human-generated texts it struggles to manage.
Proprietary tools are often out of reach for academic researchers who want to investigate the technology’s issues – both in terms of access to a model’s underlying code and possessing resources to build and train their own language models. Meta’s latest code release, however, can help them study these systems in more detail.
“We are sharing Open Pretrained Transformer, a language model with 175 billion parameters trained on publicly available data sets, to allow for more community engagement in understanding this foundational new technology,” researchers at the social media biz said on Tuesday. “For the first time for a language technology system of this size, the release includes both the pretrained models and the code needed to train and use them.”
Meta has also released subsets of the full model – up to 66 billion parameters – for anyone to use. The complete and largest OPT-175 system, however, is only available to researchers on request for noncommercial applications. It was trained using 992 Nvidia 80GB A100 GPUs, reaching a performance of 147 TFLOPS per chip. Future researchers won’t need to build the model and train it from scratch, because Meta is providing them with the code to deploy it on 16 Nvidia V100 GPUs.
Training such large models is tricky. Meta’s team of researchers said they experienced numerous failures, and had to restart the whole process 35 times over a two month period, according to a paper [PDF] on arXiv.
A Meta spokesperson told The Register releasing OPT-175 will help academics reproduce results from large language model (LLM) papers.
“It is important to improve transparency and openness around large-scale research so that the future we build with this technology is more equitable and fair. The future of LLM work cannot solely live in the hands of those with financial interests in keeping this research behind closed doors,” the spokesperson stated.
The code for Meta’s smaller pre-trained models can be found here. If you’re an academic and want the full version you can request it by completing this form. ®
Emperor Penguin Linus Torvalds has released the first release candidate for Linux 6.0, but doesn’t mind what you call it.
“After I had already decided to call this kernel 6.0, a few Chinese developers piped up and pointed out that ‘5.20’ is a more wholesome version of the Western ‘4.20’ internet-famous number,” he wrote in his announcement that Linux 6.0 rc1 has been released.
“4.20” is a reference to a day on which some celebrate marijuana, while “5.20” does likewise for magic mushrooms.
“So if you want to call this ‘Linux 5.20’, go right ahead,” Torvalds wrote.
“Because the kernel version numbers really are entirely made up and have no intrinsic meaning.”
That this week’s release has the 6.0 label is still nice to know, as discussion on the Linux kernel mailing list in recent weeks used 5.20 and 6.0 interchangeably.
As The Register has already reported, the release does not make major changes to the kernel but does include many useful updates – such as more RISC-V support, code to drive Intel’s Gaudi accelerators, and improved ACPI handling.
Torvalds lamented some Rust-enabling code didn’t make it into the release.
“I actually was hoping that we’d get some of the first rust infrastructure, and the multi-gen LRU VM, but neither of them happened this time around,” he mused, before observing “There’s always more releases.”
“This is one of those releases where you should not look at the diffstat too closely, because more than half of it is yet another AMD GPU register dump,” he added, noting that Intel’s Gaudi2 Ai processors are also likely to produce plenty of similar kernel additions.
“The CPU people also show up in the JSON files that describe the perf events, but they look absolutely tiny compared to the ‘asic_reg’ auto-generated GPU and AI hardware definitions,” he added.
The release includes 13,099 changed files, 1,280,295 insertions and 341,210 deletions. Torvalds calculated those numbers “just because I was curious and looked.”
He wants you to be curious too – or at least curious enough to test the kernel, because that’s what release candidates are for and this one contains at least one active bug. ®
Ireland is one of 19 countries worldwide that strongly dislikes Tinder. One in five Tweets by Irish people about all apps are negative.
According to Electronics Hub’s analysis of the most hated apps in the world, Tinder is the most loathed app in Ireland.
Irish people are not alone in their hatred for the dating app. Tinder was the most hated app in 19 countries in total, with Canadians, Americans, Nigerians, Kenyans and our neighbours in the UK also singling it out as their least favourite.
Electronics Hub determined the most hated apps in each country by analysing Twitter data. It processed more than 3m geotagged tweets related to 87 social media, dating, mobile games, entertainment, cryptocurrency and money transfer apps.
Researchers calculated the percentage of tweets about each app that were negative using a sentiment analysis tool which identifies whether a tweet has positive, negative or neutral sentiment.
Click to enlarge and see the most hated apps in the world by country. Infographic: Electronics Hub
Ireland was found to be one of the most negative countries when it came to attitudes towards apps. One in five Tweets posted by Irish people about apps were negative, Electronics Hub found.
Despite Irish people’s professed loathing for Tinder, the dating platform tried to play a role in keeping daters safe in the pandemic. It hooked up with the HSE to promote vaccines by adding badges to users’ profiles.
Tinder was only the second-most hated app in the world, with Roblox taking first place. More than 20 countries said the child-targeted gaming app was their most hated app. Other unpopular apps include Snapchat, Disney and Reddit.
Neighbouring countries tend to dislike similar apps, with the Scandinavians professing a dislike for Reddit and South Americans hating e-commerce apps.
Dating apps, meanwhile, are disliked the world over. In Iraq, 71.4pc of all tweets about Tinder are negative, which is the highest out of any country. A state-by-state breakdown of the most hated apps in North America also found Tinder took the top spot in 21 states.
10 things you need to know direct to your inbox every weekday. Sign up for the Daily Brief, Silicon Republic’s digest of essential sci-tech news.
Amazon workers say they are working in a “sweatshop” as safety concerns and worries about the cost of living crisis have triggered walkouts at warehouses around the country.
The Observer has spoken to four staff involved in the walkouts, who work at three Amazon warehouses, including Tilbury in Essex, where protests began on 4 August. All say they will struggle to survive this winter with pay rise offers between 35p and 50p an hour – far less than the rate of inflation, which is currently at 9.4%.
The workers, who spoke anonymously for fear of reprisals from Amazon, said they were speaking out to highlight how the firm’s ultra-cheap, ultra-convenient, super-fast delivery model works.
Amazon employs more than 70,000 people in the UK, adding 25,000 staff in 2021 alone. Many work at the company’s 21 fulfilment centres, where some workers say they are asked to carry out long, physical shifts, with difficult targets, for low pay.
Starting pay in Amazon warehouses will shortly be increasing to between £10.50 and £11.45 per hour, depending on location. An Amazon spokesperson saidthis was a 29% increase in the minimum hourly wage paid to staff since 2018. They said it is also augmented by a comprehensive benefits package worth thousands of pounds a year, and a company pension plan.
But staff say it is too low for the type of work being done and given the current economic crisis, especially at a company that just posted $121bn (£100bn) in revenues in the second quarter of 2022 alone.
“When we heard the news, it was shocking,” said one worker at Amazon’s warehouse in Tilbury. “It’s ridiculous. Inflation is [forecast to reach] 13%, and our salary increases barely 3%.” The worker rents a house with her husband for £1,350 a month without bills. “My salary is £1,600. … I’m lucky I’m married, otherwise I’d be homeless.”
Some staff are seeking a pay rise of £2 an hour from the tech giant.
Another worker at Amazon’s warehouse in Tilbury said they were “petrified” about how they would survive this winter. “We had a scenario recently where someone was living in [an] Amazon [warehouse],” he said. “If I’m honest, I can probably see that happening again.
“I can see people staying in the canteen all the time because they can’t afford to go home.”
The worker is protesting against the poor pay offer, as well as conditions that lock staff in cages for entire shifts at the warehouses, from where they pick items to be delivered to customers. (Amazon says the workstations are to protect workers from moving robotics.)
“It’s a Chinese sweatshop in the UK,” said the second worker at Tilbury. “It’s how they set up their model.”
The worker has struggled with his mental health while working for the company. “I’ve realised how bad Amazon is for my mental health,” he said. “The anxiety of going into work, knowing you’ve got to do the same stuff day in, day out, is horrible.”
That concern is echoed by a worker at an Amazon facility near Bristol, who has worked there with his wife for three years. “It was good initially,” the worker said. “There was a lot of safety consciousness, and the targets were pretty reasonable. But now they’re just pushing it higher and higher, and exploiting people.”
Around 100 Amazon staff at Bristol staged a sit-in at the company canteen on 10 August – action for which they say they were docked pay by management at the site. “The vast majority of people went back to work at that point, because at the end of the day, as much as they want to fight for it, they have to think about themselves financially.”
The Bristol warehouse worker says that managers used to stop employees from lifting heavy items from bins on high shelves in the warehouse without a ladder. “If you overstretched yourself for 10 hours, you’d end up with a bad neck and a bad back,” he said.
That has subsequently changed as staff said they felt pressured to meet ever-escalating demand. Staff pushing carts around the warehouse used to be limited to using one cart at a time for safety reasons; now it is claimed managers turn a blind eye to staff pulling two carts at once. “They don’t say nothing because all they care about is getting the work done as fast as possible,” he said. “Safety just goes out the window.”
He says he has personally lifted items weighing up to 25kg by himself, despite rules saying anything heavier than 15kg should be lifted by two people.
A worker at an Amazon facility in the north-west of Englandsaid that managers at his warehouse similarly ignored rules around not running on site and lifting down heavy items from high areas in an attempt to meet targets, which at his site require two items to be picked every minute.
Amazon declined to respond to specific claims.
Martha Dark, director at Foxglove, a non-profit organisation working to highlight issues within tech companies that supports Amazon workers, said: “None of the workers we’re supporting wanted to protest.
“They’re desperate and can’t survive on these wages. Meanwhile, Amazon threatens to dock pay and send workers to HR for revealing the truth about life in the warehouse.”
She added: “Amazon needs to respect workers’ rights to organise, stop penalising people who are fighting to survive and provide a real pay rise now.”
Two workers said they plan to leave the company because of the conditions and pay. However, some hope to stay put – to change things.
“If a lot of us who are experienced leave Amazon at this point they’ll get a new group of people in who they can mould into this depressing way of work,” said the Bristol worker. “That’s the problem.”