Delta airlines, with $40B of revenue last year, has less than 8 hours of backup power for their mission critical computers?
> According to the flight captain of JFK-SLC this morning, a routine scheduled switch to the backup generator this morning at 2:30am caused a fire that destroyed both the backup and the primary. Firefighters took a while to extinguish the fire. Power is now back up and 400 out of the 500 servers rebooted, still waiting for the last 100 to have the whole system fully functional.
http://www.flyertalk.com/forum/27032000-post135.html
Update: Confirmed.
> But Georgia Power said the issue had to do with Delta's own equipment, not a larger power outage. "We believe Delta Air Lines experienced an equipment outage; other Georgia Power customers were not affected," said John Kraft, spokesperson for the utility. Georgia power has staff on site trying to assist Delta, he said.
https://www.washingtonpost.com/news/the-switch/wp/2016/08/08...
Plus sometimes backup power isn't reliable (e.g. generator problem) or they have different tiers of coverage (and Delta didn't pay for it). We'll just have to wait and see.
There are things that are fairly easy to make redundant. I don't know anything about their infrastructure, but I doubt building a failover would be trivial.
Even if you had it in place, the risk of something occuring as you switch back and forth might not be worth it if you expect your regular setup to come back up within a reasonable time frame. Most people around here know how close to impossible it is to replicate real world usage on a test system, and how much overhead it adds to any change you make to your production system if you want the others to follow.
I've worked at several places where there has been battery backup, diesel generators, and a a disaster recovery site with generally nice and thorough strategies for replication including really expensive "sysplex-mode" for mainframes, topped with regular contigency tests where one half of the site is shut down.
And somehow, at every place, eventually something bad happened in a way that took down the applications.
It could be a power outage in the exactly the wrong circuit, fluctuating power messed up exactly the wrong storage system, someone dug up exactly the wrong fiber, the fire extinguisher set off by mistake and sprayed just the wrong cabinet with water fog through a door(!) that was supposed to automatically close, the diesel generator caught fire and the backup battery had to be taken offline, etc etc.
I'm not saying that you shouldn't try, but it's damn hard.
One day during a power cut the generator kicked in as it was supposed to, but a minute later it spluttered to a halt. After much fiddling around by maintenance staff they gave up and we were all told to go home.
The postmortem revealed that the diesel had become contaminated with all sorts of gunk - water, grit/sand and lumps of unidentifiable "stuff". The best laid plans and all that.
Edit: And cigarette butts as well.
How often did you test that?
> has less than 8 hours of backup power for their mission critical computers?
It's plausible that they thought they had that, but some part of it failed.
Not parent, but here is a single data point. When I was a student, I worked for some sysadmins who managed a small, internal data center. ~200 IBM servers, a SAN, an old VAX, etc. It had a backup gas generator that was tested loudly, every single week. It was also tested whenever the power went out for too long.
Obviously lots can go wrong, even with the best plans, but at least at the tiny little place I worked, the generator was tested every week.
Mind you, I'm not saying they don't have one, just that it can be difficult to get -- there can also be limits on the number of hours you can run them.
Even on classic DBs, setting up a failover with a lag of 15 minutes isn't a herculean effort. It's expensive, yes, but probably worth it...