Connect with us

Technology

Excuse me, what just happened? Resilience is tough when your failure is due to a ‘sequence of events that was almost impossible to foresee’

Voice Of EU

Published

on

Feature When designing systems that our businesses will rely on, we do so with resilience in mind.

Twenty-five years ago, technologies like RAID and server mirroring were novel and, in some ways, non-trivial to implement; today this is no longer the case and it is a reflex action to procure multiple servers, LAN switches, firewalls, and the like to build resilient systems.

This does not, of course, guarantee us 100 per cent uptime. The law of Mr Murphy applies from time to time: if your primary firewall suffers a hardware failure, there is a tiny, but non-zero, chance that the secondary will also collapse before you finish replacing the primary.

If you have a power failure, there is a similarly micro-tangible likelihood that the generator you have tested weekly for years will choose this moment to cough stubbornly rather than roaring into life. Unless you are (or, more accurately, the nature of your business is) so risk-averse that you can justify spending on more levels of resilience to reduce the chance of an outage even further (but never, of course, to nothing).

There are occasions, though, where planning for failure becomes hard.

Let us look at a recent example. In July 2020, the main telco in Jersey had a major outage because of a problem with a device providing time service to the organisation’s network. The kicker in this event was that the failed device did not fail in the way we are all used to – by making a “bang” noise and emitting smoke; had it done so, in fact, all would have been well as the secondary unit would have taken over.

Impossible

No, this was a more devious kind of time server which only part-failed. It kept running but started serving times from about 20 years in the past (by no coincidence at all this was the factory default time setting), thus confusing network infrastructure devices and causing traffic to stop flowing.

Customer dissatisfaction was palpable, of course, but as an IT specialist one does have to feel something for the company’s technical team: how many of us would ever consider, as a possible failure case, something that the technical chief described quite correctly as a “sequence of events that was almost impossible to foresee”?

(Incidentally, in a somewhat more good-news story, stepping back a moment to our point about extra layers of resilience, the same company had previously survived three offshore cables being severed… by having a fourth).

Could monitoring tools have been put in place to see issues like this when they happen? Yes, absolutely, but the point is that to do so one would first need to identify the scenarios as something that could happen. In the sense of risk management, this type of failure – very high impact but infinitesimally unlikely – is the worst possible kind for a risk manager. There are theories and books about how one can contemplate and deal with such risks, the best-known of which is probably Nassim Nicholas Taleb’s book The Black Swan, which talks of just this kind of risk, but if you want to try to defend against the unexpected then at the very least you need to sit down with a significant number of people in a highly focused way, preferably with an expert in the field to guide and moderate, and work on identifying such possible “black swan” events.

While the black swan concept is most definitely a thing to bear in mind, there is in fact a far more common problem with systems that we consider resilient – a failure to understand how the resilience works.

One particular installation at a company with an office and two data centres had point-to-point links in a triangle between each premises, and each data centre had an internet connection. The two firewalls, one in each data centre, were configured as a resilient pair, and worked as such for years. One day internet service went down, and investigation showed that the secondary unit had lost track of the primary and had switched itself to become the primary. Having two active primaries caused split traffic flows, and hence an outage.

Predictable

In hindsight, this was completely predictable. The way the primary/secondary relationship was maintained between the devices was for the primary to send a “heartbeat” signal to the secondary every few seconds; if the secondary failed to receive the heartbeat three times, it woke up and acted as a primary. Because the devices were in separate data centres, they were connected through various pieces of technology: a LAN patch cord into a switch, into a fibre transceiver, into a telco fibre, then the same in reverse at the other end.

A fault on any one of those elements could cause the network devices to reconfigure their topology to switch data over the other way around the fibre triangle – with the change causing a network blip sufficiently long to drop three heartbeats. In fact, the only approved configuration for the primary/secondary interconnection was a crossover Ethernet cable from one device to the other: the failover code was written with the assumption that, aside perhaps from a highly unlikely sudden patch cord fault, the primary becoming invisible to the secondary meant that the former had died.

Many of us have come across similar instances, where something we expected to fail over has not done so. It’s equally common, too, to come across instances where the failover works OK but then there are issues with the failback, which can be just as problematic. I recall a global WAN I once worked on where, for whatever reason, failovers from primary to secondary were so quick that you didn’t notice any interruption (the only clue was the alert from the monitoring console) but there was a pause of several seconds when failing back.

In the firewall example, even when connectivity was restored the devices would not re-synch without a reboot: remember, the only supported failure scenario was the primary dying completely, which meant that it was only at boot time that it would check to see which role its partner was playing so it could act accordingly. Until someone turned it off and back on again, there was no chance that the problem would go away.

To make our resilient systems truly resilient, then, we need to do three things.

First, we should give some thought to those “black swan” events. It may be that we cannot afford masses of time and effort to consider such low-probability risks, but at the very least we should take a conscious decision on how much or how little we will do in that respect: risk management is all about reasoning and making conscious decisions like that.

Expertise

Second, if we don’t have the knowledge of the precise way our systems and their failover mechanisms work, we must engage people who do and get the benefit of their expertise and experience… and while we’re at it, we should read the manual: nine times out of ten it will tell us how to configure things, even if it doesn’t explain why.

Finally, though, we need to test things – thoroughly and regularly. In our firewall example all potential failure modes should have been considered: if a failure of one of a handful of components could cause an outage, why not test all of them? And when we test, we need to do it for real: we don’t just test failover in the lab and then install the kit in a production cabinet, we test it once it’s in too.

This may need us to persuade the business that we need downtime – or at least potential downtime to cater for the test being unsuccessful – but if management have any sense, they will be persuadable that an approved outage during a predictable time window with the technical team standing by and watching like hawks is far better than an unexpected but entirely foreseeable outage when something breaks for real and the resilience turns out not to work.

Testing

Oh, and when you test failover and failback, run for several days in a failed-over state if you can: many problems don’t manifest instantly, and you will always learn more in a multi-day failover than in one that lasts only a couple of minutes. Bear in mind also the word “regularly” that I used alongside “thoroughly”. Even if we know there has been no change to a particular component, there may well be some knock-on effect from a change to something else. Something that used to be resilient may have become less resilient or even non-resilient because something else changed and we didn’t realise the implication – so regular resilience testing is absolutely key.

Because if something isn’t resilient, this will generally not be because of some esoteric potential failure mode that is next to impossible to anticipate and/or difficult or impossible to test. Most of the time it will because something went wrong – or something was configured wrongly – in a way you could have emulated in a test. ®

Source link

Technology

How scientists in Ireland are using technology to predict the climate

Voice Of EU

Published

on

Scientists at ICHEC have used supercomputing to predict Ireland’s weather patterns for the rest of the century.

In August, the Intergovernmental Panel on Climate Change (IPCC) spelled out the intensity of the climate crisis affecting every region of the world because of human activity, and Ireland is no exception.

Scientists at the Irish Centre for High-End Computing (ICHEC), based at NUI Galway, have been using advanced technology to create climate models and simulations that indicate the impact of the climate crisis on Ireland by mid-century.

Their work has raised some concerning predictions for Ireland’s weather patterns in the coming decades, including more heatwaves, less snow, and increasingly unpredictable rainfall patterns – even by Irish standards.

Temperatures are set to increase by between one and 1.6 degrees Celsius relative to levels experienced between 1991 to 2000, with the east seeing the sharpest rise. Heatwaves, especially in the south-east of the country, are expected to become more frequent.

The simulations also found that the number of days Ireland experiences frost and ice will be slashed by half, as will the amount of snow that falls in winter. Rainfall will be more variable with longer dry and wet periods, and surface winds will become weaker.

‘Dramatic changes’

While the report suggests that a heating climate may be good for farming in Ireland – a significant contributor to the economy – it will also be accompanied by the rise of pests that can have potentially devastating effects on agriculture.

Reduced wind strength and unpredictable weather will have an impact on Ireland’s growing renewable energy infrastructure, which relies heavily on specific climate conditions to reach targets.

“A mean warming of two or three degrees Celsius does not seem like much, given that temperatures can vary by a lot more than that just from day to day,” said ICHEC climate scientists Dr Paul Nolan and Dr Enda O’Brien.

“However, even that amount of warming is likely to lead to widespread and even dramatic changes in ice cover – especially in the Arctic – to sea levels, and in the natural world of plants and animals.”

Ireland’s contribution

With machine learning and supercomputing, scientists are able to use historical climate data and observations to improve predictions of Earth’s future climate – and the impacts of the climate crisis.

Ireland is part of a consortium of several northern European countries that contribute to the IPCC reports by running global climate models that feed into the report’s assessment.

As part of the consortium, Nolan has conducted many centuries worth of global climate simulations using the EC-Earth climate model, which represents the most relevant physical processes that operate in the atmosphere, oceans, land surface and sea ice.

The simulations range from historical data – so the model can be compared to real climate records – to the end of the 21st century, with the aim of providing a comprehensive picture of climate trends and what the future could hold. The ICHEC research is funded and supported by the Environmental Protection Agency, Met Éireann and the Marine Institute in Galway.

“The level of detail and consistency achieved gives confidence in these projections and allows an ever more persuasive evidence-based consensus to emerge that humans are forcing rapid climate change in well-understood ways,” Nolan and O’Brien wrote in the Irish Times this week.

“How to respond to that consensus now is a matter primarily for governments, since they can have the most impact, as well as for individuals.”

Don’t miss out on the knowledge you need to succeed. Sign up for the Daily Brief, Silicon Republic’s digest of need-to-know sci-tech news.

Source link

Continue Reading

Technology

Apple’s plan to scan images will allow governments into smartphones | John Naughton

Voice Of EU

Published

on

For centuries, cryptography was the exclusive preserve of the state. Then, in 1976, Whitfield Diffie and Martin Hellman came up with a practical method for establishing a shared secret key over an authenticated (but not confidential) communications channel without using a prior shared secret. The following year, three MIT scholars – Ron Rivest, Adi Shamir and Leonard Adleman – came up with the RSA algorithm (named after their initials) for implementing it. It was the beginning of public-key cryptography – at least in the public domain.

From the very beginning, state authorities were not amused by this development. They were even less amused when in 1991 Phil Zimmermann created Pretty Good Privacy (PGP) software for signing, encrypting and decrypting texts, emails, files and other things. PGP raised the spectre of ordinary citizens – or at any rate the more geeky of them – being able to wrap their electronic communications in an envelope that not even the most powerful state could open. In fact, the US government was so enraged by Zimmermann’s work that it defined PGP as a munition, which meant that it was a crime to export it to Warsaw Pact countries. (The cold war was still relatively hot then.)

In the four decades since then, there’s been a conflict between the desire of citizens to have communications that are unreadable by state and other agencies and the desire of those agencies to be able to read them. The aftermath of 9/11, which gave states carte blanche to snoop on everything people did online, and the explosion in online communication via the internet and (since 2007) smartphones, has intensified the conflict. During the Clinton years, US authorities tried (and failed) to ensure that all electronic devices should have a secret backdoor, while the Snowden revelations in 2013 put pressure on internet companies to offer end-to-end encryption for their users’ communications that would make them unreadable by either security services or the tech companies themselves. The result was a kind of standoff: between tech companies facilitating unreadable communications and law enforcement and security agencies unable to access evidence to which they had a legitimate entitlement.

In August, Apple opened a chink in the industry’s armour, announcing that it would be adding new features to its iOS operating system that were designed to combat child sexual exploitation and the distribution of abuse imagery. The most controversial measure scans photos on an iPhone, compares them with a database of known child sexual abuse material (CSAM) and notifies Apple if a match is found. The technology is known as client-side scanning or CSS.

Powerful forces in government and the tech industry are now lobbying hard for CSS to become mandatory on all smartphones. Their argument is that instead of weakening encryption or providing law enforcement with backdoor keys, CSS would enable on-device analysis of data in the clear (ie before it becomes encrypted by an app such as WhatsApp or iMessage). If targeted information were detected, its existence and, potentially, its source would be revealed to the agencies; otherwise, little or no information would leave the client device.

CSS evangelists claim that it’s a win-win proposition: providing a solution to the encryption v public safety debate by offering privacy (unimpeded end-to-end encryption) and the ability to successfully investigate serious crime. What’s not to like? Plenty, says an academic paper by some of the world’s leading computer security experts published last week.

The drive behind the CSS lobbying is that the scanning software be installed on all smartphones rather than installed covertly on the devices of suspects or by court order on those of ex-offenders. Such universal deployment would threaten the security of law-abiding citizens as well as lawbreakers. And even though CSS still allows end-to-end encryption, this is moot if the message has already been scanned for targeted content before it was dispatched. Similarly, while Apple’s implementation of the technology simply scans for images, it doesn’t take much to imagine political regimes scanning text for names, memes, political views and so on.

In reality, CSS is a technology for what in the security world is called “bulk interception”. Because it would give government agencies access to private content, it should really be treated like wiretapping and regulated accordingly. And in jurisdictions where bulk interception is already prohibited, bulk CSS should be prohibited as well.

In the longer view of the evolution of digital technology, though, CSS is just the latest step in the inexorable intrusion of surveillance devices into our lives. The trend that started with reading our emails, moved on to logging our searches and our browsing clickstreams, mining our online activity to create profiles for targeting advertising at us and using facial recognition to allow us into our offices now continues by breaching the home with “smart” devices relaying everything back to motherships in the “cloud” and, if CSS were to be sanctioned, penetrating right into our pockets, purses and handbags. That leaves only one remaining barrier: the human skull. But, rest assured, Elon Musk undoubtedly has a plan for that too.

What I’ve been reading

Wheels within wheels
I’m not an indoor cyclist but if I were, The Counterintuitive Mechanics of Peloton Addiction, a confessional blogpost by Anne Helen Petersen, might give me pause.

Get out of here
The Last Days of Intervention is a long and thoughtful essay in Foreign Affairs by Rory Stewart, one of the few British politicians who always talked sense about Afghanistan.

The insider
Blowing the Whistle on Facebook Is Just the First Step is a bracing piece by Maria Farrell in the Conversationalist about the Facebook whistleblower.

Source link

Continue Reading

Technology

Criminals use fake AI voice to swindle UAE bank out of $35m • The Register

Voice Of EU

Published

on

In brief Authorities in the United Arab Emirates have requested the US Department of Justice’s help in probing a case involving a bank manager who was swindled into transferring $35m to criminals by someone using a fake AI-generated voice.

The employee received a call to move the company-owned funds by someone purporting to be a director from the business. He also previously saw emails that showed the company was planning to use the money for an acquisition, and had hired a lawyer to coordinate the process. When the sham director instructed him to transfer the money, he did so thinking it was a legitimate request.

But it was all a scam, according to US court documents reported by Forbes. The criminals used “deep voice technology to simulate the voice of the director,” it said. Now officials from the UAE have asked the DoJ to hand over details of two US bank accounts, where over $400,000 from the stolen money were deposited.

Investigators believe there are at least 17 people involved in the heist.

AI systems need to see the human perspective

Facebook has teamed up with 13 universities across nine countries to compile Ego4D, a dataset containing more than 2,200 hours of video shot in first-person, where 700 participants were filmed performing everyday activities like cooking or playing video games.

The antisocial network is hoping Ego4D will unlock new capabilities in augmented and virtual reality or robotics. New models trained on this data can be tested on a range of tasks, including episodic memory, predicting what happens next, coordinating hand movement to manipulate objects, and social interaction.

“Imagine your AR device displaying exactly how to hold the sticks during a drum lesson, guiding you through a recipe, helping you find your lost keys, or recalling memories as holograms that come to life in front of you,” Facebook said in a blog post.

“Next-generation AI systems will need to learn from an entirely different kind of data – videos that show the world from the center of the action, rather than the sidelines,” added Kristen Grauman, lead research scientist at Facebook.

Researchers will have access to Ego4D later next month subject to a data use agreement.

Microsoft Translator’s AI software

Microsoft Translator, language translation software powered by neural networks, can now translate over 100 different languages.

Twelve new languages and dialects were added to Microsoft Translator this week, including: endangered ones like Bashkir spoken by a Kipchak Turkic ethnic group indigenous to Russia to more common lingos like Mongolian. Microsoft Translator now supports 103 languages.

“One hundred languages is a good milestone for us to achieve our ambition for everyone to be able to communicate regardless of the language they speak,” said Xuedong Huang, Microsoft technical fellow and Azure AI chief technology officer.

Xuedong said the software is based on a multilingual AI model called Z-code. The system deals with text, and is part of Microsoft’s efforts to build a larger multimodal system capable of handling images, text, and audio dubbed the XYZ-code vision. Microsoft Translator is deployed in a range of services, including search engine Bing and offered as an API on its cloud platform Azure Cognitive Services.

ShotSpotter sues Vice for defamation and wants $300m in damages

The controversial AI gunshot-detection company Shotspotter has sued Vice, claiming its business has been unfairly tarnished by a series of articles published by the news outlet.

“On July 26, 2021, Vice launched a defamatory campaign in which it falsely accused ShotSpotter of conspiring with police to fabricate and alter evidence to frame Black men for crimes they did not commit,” the complaint said.

ShotSpotter accused the publication of portraying the company’s technology and actions inaccurately to “cultivate a ‘subversive’ brand” used to sell products advertised in its “sponsored content”.

The company made headlines when evidence used to try to prove a Black man shot and killed another man in a court trial was retracted. The defense lawyer accused ShotSpotter employees of tampering with the evidence to support the police’s case. Vice allegedly made false claims that the biz routinely used its software to tag loud sounds as gunshots to help law enforcement prosecute innocent suspects in shooting cases.

When Vice’s journalists were given proof to show that wasn’t the case, they refused to correct their factual inaccuracies, the lawsuit claimed. ShotSpotter argued the articles had ruined its reputation and now it wants Vice to cough up a whopping $300m in damages.

State of AI 2021

The annual State of AI report is out, compiled by two British tech investors, recapping this year’s trends and developments in AI.

The fourth report from Nathan Benaich, a VC at Air Street Capital, and Ian Hogarth, co-founder of music app Songkick and an angel investor, focuses on transformers, a type of machine learning architecture best known for powering giant language models like OpenAI’s GPT-3 or Google’s BERT.

Transformers aren’t just useful for generating text; they’ve proven adept in other areas, like computer vision or biology too. Machine learning technology is also continuing to mature – developers are deploying more systems to tackle real-world problems such as optimising energy through national electric grids or warehouse logistics for supermarkets.

That also applies to military applications, the pair warned. “AI researchers have traditionally seen the AI arms race as a figurative one – simulated dogfights between competing AI systems carried out in labs – but that is changing with reports of recent use of autonomous weapons by various militaries.”

You can read the full report here. ®

Source link

Continue Reading

Trending

Subscribe To Our Newsletter

Join our mailing list to receive the latest news and updates 
directly on your inbox.

You have Successfully Subscribed!