Why Google's London backup generators never started
Batteries held the site for five minutes. Then the controller that should have started the generators decided it didn't need to.
On 20 October, a power failure at Google's London cloud site disrupted customers including Lloyds, HSBC, Royal Mail and Revolut, according to The Register. This week Google explained what went wrong. The cause was not a lack of backup. It was the logic that decides when to use it.
The site, a zone called europe-west2-b, normally takes power from two separate utility feeds, A and B. On the day, feed B was switched off. Its breaker (a switch that cuts a circuit) had been opened for repairs after problems showed up in a generator load test on 14 October. Then an underground cable on feed A broke. Both feeds were dead.
The site's uninterruptible power supplies (UPS, banks of batteries) took over. They were designed to carry the load for five minutes, and they did. The diesel generators were supposed to start inside that window. They never did.
Google says the cause was a design flaw in how the site senses voltage, together with a misconfiguration in its programmable logic controller (PLC, a small industrial computer that runs the switching). The controller's logic concluded that a healthy utility supply was available, even though both feeds were de-energised. So it never sent the command to start the generators.
The outage began just after 07:05 BST. Google said the power event was mitigated by 12:27, and most workloads were back by 14:48, according to the UK IT firm ServNet. Google says it reprogrammed the controller once the utility was reconnected and has changed the site's electrical set-up and tested it. It also plans wider testing of infrastructure-level failures and faster notices to customers.
Why it matters to youBackup only works if the system that switches to it is right. Here, two feeds, batteries and generators were all in place, and one logic error defeated them. When you design a backup, ask what decides when it starts, and what that decision can't see.