It is a technology that has been frowned upon by ethicists: now researchers are hoping to unmask the reality of emotion recognition systems in an effort to boost public debate.
Technology designed to identify human emotions using machine learning algorithms is a huge industry, with claims it could prove valuable in myriad situations, from road safety to market research. But critics say the technology not only raises privacy concerns, but is inaccurate and racially biased.
A team of researchers have created a website – emojify.info – where the public can try out emotion recognition systems through their own computer cameras. One game focuses on pulling faces to trick the technology, while another explores how such systems can struggle to read facial expressions in context.
Their hope, the researchers say, is to raise awareness of the technology and promote conversations about its use.
“It is a form of facial recognition, but it goes farther because rather than just identifying people, it claims to read our emotions, our inner feelings from our faces,” said Dr Alexa Hagerty, project lead and researcher at the University of Cambridge Leverhulme Centre for the Future of Intelligence and the Centre for the Study of Existential Risk.
But Hagerty said many people were not aware how common emotion recognition systems were, noting they were employed in situations ranging from job hiring, to customer insight work, airport security, and even education to see if students are engaged or doing their homework.
Such technology, she said, was in use all over the world, from Europe to the US and China. Taigusys, a company that specialises in emotion recognition systems and whose main office is in Shenzhen, says it has used them in settings ranging from care homes to prisons, while according to reports earlier this year, the Indian city of Lucknow is planning to use the technology to spot distress in women as a result of harassment – a move that has met with criticism, including from digital rights organisations.
While Hagerty said emotion recognition technology might have some potential benefits these must be weighed against concerns around accuracy, racial bias, as well as whether the technology was even the right tool for a particular job.
“We need to be having a much wider public conversation and deliberation about these technologies,” she said.
The new project allows users to try out emotion recognition technology. The site notes that “no personal data is collected and all images are stored on your device”. In one game, users are invited to pull a series of faces to fake emotions and see if the system is fooled.
“The claim of the people who are developing this technology is that it is reading emotion,” said Hagerty. But, she added, in reality the system was reading facial movement and then combining that with the assumption that those movements are linked to emotions – for example a smile means someone is happy.
“There is lots of really solid science that says that is too simple; it doesn’t work quite like that,” said Hagerty, adding that even just human experience showed it was possible to fake a smile. “That is what that game was: to show you didn’t change your inner state of feeling rapidly six times, you just changed the way you looked [on your] face,” she said.
Some emotion recognition researchers say they are aware of such limitations. But Hagerty said the hope was that the new project, which is funded by Nesta (National Endowment for Science, Technology and the Arts), will raise awareness of the technology and promote discussion around its use.
“I think we are beginning to realise we are not really ‘users’ of technology, we are citizens in world being deeply shaped by technology, so we need to have the same kind of democratic, citizen-based input on these technologies as we have on other important things in societies,” she said.
Vidushi Marda, senior programme officer at the human rights organisation Article 19 said it was crucial to press “pause” on the growing market for emotion recognition systems.
“The use of emotion recognition technologies is deeply concerning as not only are these systems based on discriminatory and discredited science, their use is also fundamentally inconsistent with human rights,” she said. “An important learning from the trajectory of facial recognition systems across the world has been to question the validity and need for technologies early and often – and projects that emphasise on the limitations and dangers of emotion recognition are an important step in that direction.”
The UK has signed up to a US plan for sharing police-held biometric data about citizens with US border officials.
According to a member of the European Parliament’s Committee on Civil Liberties, Justice and Home Affairs (LIBE), the body met “informally” with representatives of the US Department of Homeland Security this week to discuss the plans.
They come under the auspices of the Enhanced Border Security Partnership (EBSP), which is designed to increase the US Department Of Homeland Security’s ability to detect threats through biometric information sharing. Israel signed up to the arrangement in March.
LIBE committee member and Pirate Party MEP Patrick Breyer said that during the meeting last week, the committee discovered that the UK – and three EU member states, though their identities were not revealed – had already signed up to reintroduce US visa requirements which grant access to police biometric databases.
In the UK, the Home Office declined the opportunity to deny it was signing up for the scheme. A spokesperson said: “The UK has a long-standing and close partnership with the USA which includes sharing data for specific purposes. We are in regular discussion with them on new proposals or initiatives to improve public safety and enable legitimate travel.”
Under UK law the police can retain an individual’s DNA profile and fingerprint record for up to three years from the date the samples were taken, even if the individual was arrested but not charged, provided the Biometrics Commissioner agrees. Police can also apply for a two-year extension. The same applies to those charged, but not convicted.
According to reports, the US Enhanced Border Security Partnership (EBSP) initiative will be voluntary initially but is set to become mandatory under the US Visa Waiver Program (VWP), which allows visa-free entry into the United States for up to 90 days, by 2027.
MEP Breyer said that when asked exactly what data the US wanted to tap into, the answer was as much as possible. When asked what would happen at US borders if a traveler was known to the police in participating states, it was said that this would be decided by the US immigration officer on a case-by-case basis.
The DHS program is part of a project to update the visa waiver scheme under which EU members and other European countries enjoy visa-free travel to the US under certain conditions.
Breyer noted: “I expect the EU Commission and also the German government to reject the demand of the US authorities and not allow themselves to be blackmailed.
“If necessary, the visa waiver program must be terminated by Europe as well. Millions of innocent Europeans are listed in police databases and could be exposed to completely disproportionate reactions in the USA.
“The US lacks adequate data and fundamental rights protection. Providing personal data to the US exposes our citizens… to the risk of arbitrary detention and false suspicion, with possible dire consequences, in the course of the US ‘war on terror’. We must protect our citizens from these practices,” Breyer said.
Google’s decision follows concerns that law enforcement could use personal data from certain apps against people who have sought abortions illegally.
Tech giant Google has said it will soon auto-delete the data of users’ visits to abortion clinics and other medical sites from their location history.
This followed the US Supreme Court’s recent decision to overturn Roe v Wade, eliminating the constitutional right to an abortion in the country.
Other medical facilities that Google mentioned in its planned location changes include counselling centres, domestic violence shelters, fertility centres, addiction treatment facilities, weight loss clinics and cosmetic surgery clinics.
The tech giant also said location history is off by default and that there are tools such as auto-delete so users can easily get rid of parts or all of their location data.
Google said the location data changes will take effect “in the coming weeks”. The tech giant also shared planned data changes around its fitness apps to protect the privacy of users.
“Fitbit users who have chosen to track their menstrual cycles in the app can currently delete menstruation logs one at a time, and we will be rolling out updates that let users delete multiple logs at once,” said Google senior VP of core systems and experiences Jen Fitzpatrick in a blog post.
Fitzpatrick said the tech giant considers the “privacy and security expectations” of people using its products and that it notifies users when it complies with legal demands for information.
“We remain committed to protecting our users against improper government demands for data, and we will continue to oppose demands that are overly broad or otherwise legally objectionable,” Fitzpatrick said.
Following the decision to overturn Roe v Wade, there have been concerns that law enforcement could use personal data from certain apps against people who have sought abortions illegally.
One type of app where this has been a concern has been period tracking apps. The Stardust app saw a recent surge in popularity in after it claimed to implement end-to-end encryption.
Last week, I missed a real-life meeting because I hadn’t set a reminder on my smartphone, leaving someone I’d never met before alone in a café. But on the same day, I remembered the name of the actor who played Will Smith’s aunt in The Fresh Prince of Bel-Air in 1991 (Janet Hubert). Memory is weird, unpredictable and, neuroscientifically, not yet entirely understood. When memory lapses like mine happen (which they do, a lot), it feels both easy and logical to blame the technology we’ve so recently adopted. Does having more memory in our pockets mean there’s less in our heads? Am I losing my ability to remember things – from appointments to what I was about to do next – because I expect my phone to do it for me? Before smartphones, our heads would have held a cache of phone numbers and our memories would contain a cognitive map, built up over time, which would allow us to navigate – for smartphone users, that is no longer true.
Our brains and our smartphones form a complex web of interactions: the smartphonification of life has been rising since the mid 2000s, but was accelerated by the pandemic, as was internet use in general. Prolonged periods of stress, isolation and exhaustion – common themes since March 2020 – are well known for their impact on memory. Of those surveyed by memory researcher Catherine Loveday in 2021, 80% felt that their memories were worse than before the pandemic. We are – still – shattered, not just by Covid-19, but also by the miserable national and global news cycle. Many of us self-soothe with distractions like social media. Meanwhile, endless scrolling can, at times, create its own distress, and phone notifications and self interrupting to check for them, also seem to affect what, how and if we remember.
So what happens when we outsource part of our memory to an external device? Does it enable us to squeeze more and more out of life, because we aren’t as reliant on our fallible brains to cue things up for us? Are we so reliant on smartphones that they will ultimately change how our memories work (sometimes called digital amnesia)? Or do we just occasionally miss stuff when we don’t remember the reminders?
Neuroscientists are divided. Chris Bird is professor of cognitive neuroscience in the School of Psychology at the University of Sussex and runs research by the Episodic Memory Group. “We have always offloaded things into external devices, like writing down notes, and that’s enabled us to have more complex lives,” he says. “I don’t have a problem with using external devices to augment our thought processes or memory processes. We’re doing it more, but that frees up time to concentrate, focus on and remember other things.” He thinks that the kind of things we use our phones to remember are, for most human brains, difficult to remember. “I take a photo of my parking ticket so I know when it runs out, because it’s an arbitrary thing to remember. Our brains aren’t evolved to remember highly specific, one-off things. Before we had devices, you would have to make a quite an effort to remember the time you needed to be back at your car.”
Professor Oliver Hardt, who studies the neurobiology of memory and forgetting at McGill University in Montreal, is much more cautious. “Once you stop using your memory it will get worse, which makes you use your devices even more,” he says. “We use them for everything. If you go to a website for a recipe, you press a button and it sends the ingredient list to your smartphone. It’s very convenient, but convenience has a price. It’s good for you to do certain things in your head.”
Hardt is not keen on our reliance on GPS. “We can predict that prolonged use of GPS likely will reduce grey matter density in the hippocampus. Reduced grey matter density in this brain area goes along with a variety of symptoms, such as increased risk for depression and other psychopathologies, but also certain forms of dementia. GPS-based navigational systems don’t require you to form a complex geographic map. Instead, they just tell you orientations, like ‘Turn left at next light.’ These are very simple behavioural responses (here: turn left) at a certain stimulus (here: traffic light). These kinds of spatial behaviours do not engage the hippocampus very much, unlike those spatial strategies that require the knowledge of a geographic map, in which you can locate any point, coming from any direction and which requires [cognitively] complex computations. When exploring the spatial capacities of people who have been using GPS for a very long time, they show impairments in spatial memory abilities that require the hippocampus. Map reading is hard and that’s why we give it away to devices so easily. But hard things are good for you, because they engage cognitive processes and brain structures that have other effects on your general cognitive functioning.”
Hardt doesn’t have data yet, but believes, “the cost of this might be an enormous increase in dementia. The less you use that mind of yours, the less you use the systems that are responsible for complicated things like episodic memories, or cognitive flexibility, the more likely it is to develop dementia. There are studies showing that, for example, it is really hard to get dementia when you are a university professor, and the reason is not that these people are smarter – it’s that until old age, they are habitually engaged in tasks that are very mentally demanding.” (Other scientists disagree – Daniel Schacter, a Harvard psychologist who wrote the seminal Seven Sins Of Memory: How The Mind Forgets and Remembers, thinks effects from things like GPS are “task specific”, only.)
While smartphones can obviously open up whole new vistas of knowledge, they can also drag us away from the present moment, like it’s a beautiful day, unexperienced because you’re head down, WhatsApping a meal or a conversation. When we’re not attending to an experience, we are less likely to recall it properly, and fewer recalled experiences could even limit our capacity to have new ideas and being creative. As the renowned neuroscientist and memory researcher Wendy Suzuki recently put it on the Huberman Lab neuroscience podcast, “If we can’t remember what we’ve done, the information we’ve learned and the events of our lives, it changes us… [The part of the brain which remembers] really defines our personal histories. It defines who we are.”
Catherine Price, science writer and author of How to Break Up With Your Phone, concurs. “What we pay attention to in the moment adds up to our life,” she says. “Our brains cannot multitask. We think we can. But any moment where multitasking seems successful, it’s because one of those tasks was not cognitively demanding, like you can fold laundry and listen to the radio. If you’re paying attention to your phone, you’re not paying attention to anything else. That might seem like a throwaway observation, but it’s actually deeply profound. Because you will only remember the things you pay attention to. If you’re not paying attention, you’re literally not going to have a memory of it to remember.”
The Cambridge neuroscientist Barbara Sahakian has evidence of this, too. “In an experiment in 2010, three different groups had to complete a reading task,” she says. “One group got instant messaging before it started, one got instant messaging during the task, and one got no instant messaging, and then there was a comprehension test. What they found was that the people getting instant messages couldn’t remember what they just read.”
Price is much more worried about what being perpetually distracted by our phones – termed “continual partial attention” by the tech expert Linda Stone – does to our memories than using their simpler functions. “I’m not getting distracted by my address book,” she says. And she doesn’t believe smartphones free us up to do more. “Let’s be real with ourselves: how many of us are using the time afforded us by our banking app to write poetry? We just passively consume crap on Instagram.” Price is from Philadelphia. “What would have happened if Benjamin Franklin had had Twitter? Would he have been on Twitter all the time? Would he have made his inventions and breakthroughs?
“I became really interested in whether the constant distractions caused by our devices might be impacting our ability to actually not just accumulate memories to begin with, but transfer them into long-term storage in a way that might impede our ability to think deep and interesting thoughts,” she says. “One of the things that impedes our brain’s ability to transfer memories from short- to long-term storage is distraction. If you get distracted in the middle of it” – by a notification, or by the overwhelming urge to pick up your phone – “you’re not actually going to have the physical changes take place that are required to store that memory.”
It’s impossible to know for sure, because no one measured our level of intellectual creativity before smartphones took off, but Price thinks smartphone over-use could be harming our ability to be insightful. “An insight is being able to connect two disparate things in your mind. But in order to have an insight and be creative, you have to have a lot of raw material in your brain, like you couldn’t cook a recipe if you didn’t have any ingredients: you can’t have an insight if you don’t have the material in your brain, which really is long term memories.” (Her theory was backed by the 92-year-old Nobel prize-winning neuroscientist and biochemist Eric Kandel, who has studied how distraction affects memory – Price bumped into him on a train and grilled him about her idea. “I’ve got a selfie of me with a giant grin and Eric looking a bit confused.”) Psychologist professor Larry Rosen, co-author (with neuroscientist Adam Gazzaley) of The Distracted Mind: Ancient Brains in a High-Tech World, also agrees: “Constant distractions make it difficult to encode information in memory.”
Smartphones are, of course, made to hijack our attention. “The apps that make money by taking our attention are designed to interrupt us,” says Price. “I think of notifications as interruptions because that’s what they’re doing.”
For Oliver Hardt, phones exploit our biology. “A human is a very vulnerable animal and the only reason we are not extinct is that we have a superior brain: to avoid predation and find food, we have had to be really good at being attentive to our environment. Our attention can shift rapidly around and when it does, everything else that was being attended to stops, which is why we can’t multitask. When we focus on something, it’s a survival mechanism: you’re in the savannah or the jungle and you hear a branch cracking, you give your total attention to that – which is useful, it causes a short stress reaction, a slight arousal, and activates the sympathetic nervous system. It optimises your cognitive abilities and sets the body up for fighting or flighting.” But it’s much less useful now. “Now, 30,000 years later, we’re here with that exact brain” and every phone notification we hear is a twig snapping in the forest, “simulating what was important to what we were: a frightened little animal.”
Smartphone use can even change the brain, according to the ongoing ABCD study which is tracking over 10,000 American children through to adulthood. “It started by examining 10-year-olds both with paper and pencil measures and an MRI, and one of their most interesting early results was that there was a relationship between tech use and cortical thinning,” says Larry Rosen, who studies social media, technology and the brain. “Young children who use more tech had a thinner cortex, which is supposed to happen at an older age.” Cortical thinning is a normal part of growing up and then ageing, and in much later life can be associated with degenerative diseases such as Parkinson’s and Alzheimer’s, as well as migraines.
Obviously, the smartphone genie is out of the bottle and has run over the hills and far away. We need our smartphones to access offices, attend events, pay for travel and to function as tickets, passes and credit cards, as well as for emails, calls and messages. It’s very hard not to have one. If we’re worried about what they – or the apps on them – might be doing to our memories, what should we do?
Rosen discusses a number of tactics in his book. “My favourites are tech breaks,” he says, “where you start by doing whatever on your devices for one minute and then set an alarm for 15 minutes time. Silence your phone and place it upside down, but within your view as a stimulus to tell your brain that you will have another one-minute tech break after the 15-minute alarm. Continue until you adapt to 15 minutes focus time and then increase to 20. If you can get to 60 minutes of focus time with short tech breaks before and after, that’s a success.”
“If you think your memory and focus have got worse and you’re blaming things like your age, your job, or your kids, that might be true, but it’s also very likely due to the way you’re interacting with your devices,” says Price, who founded Screen/Life Balance to help people manage their phone use. As a science writer, she’s “very much into randomly controlled trials, but with phones, it’s actually more of a qualitative question about personally how it’s impacting you. And it’s really easy to do your own experiment and see if it makes a difference. It’s great to have scientific evidence. But we can also intuitively know: if you practice keeping your phone away more and you notice that you feel calmer and you’re remembering more, then you’ve answered your own question.”