Hacker News Reader

Comment by dholowiski | Hacker News Reader

I worked in a datacenter (serving many companies you've heard of) during a catastrophic power failure that lasted almost 24 hours. It's kind of like a plane crash - it's never one thing failing that causes the problem (that's what redundancy is for) - it's that perfect chain of events, multiple 'once in a lifetime' failures that causes it.

For example, power outage occurs at the same time the UPS batteries are being changed. Bypass fails and Diesel generators fail to kick in. Circuit breakers blow everywhere making it extremely difficult to get the generators back on line. This all happens in the middle of the night in a winter storm (or 'lightning storm) which causes a further delay in response time.

Been there, done that.

Edit: Also, bureaucracy, lack of documentation, and a manager CF added several hours to the outage. Sometimes you just have to STFU and let the geeks fix the problem.

View on HN

Except, this isn't a once in a lifetime occurrence. As of today it's twice in a lifetime. That's indicates to me they didn't just hit that perfect chain of events, it tells me they have problems in their architecture or implementation that need to be addressed and tested before the next crisis occurs.

I worked in a datacenter as well. The whole point is that you test those scenarios when it's not an emergency so you can be confident everything will operate smoothly.

Don't discount the chance that this was caused by two totally different 'once in a lifetime' chain of events... i've seen it happen.

However, as you point out it is far more likely that there are architecture or implementation issues that caused the repeated outages.

There are so many reasons that I don't want to have servers in California... high costs, dense population vulnerability to earthquakes and a hot terrorism target. Massive bandwidth but massive utilization, somewhat shaky power grid, heck even the state has financial issues. I picked Atlanta for my Linode server which is better (it's still a little close to the coast for me) and I host my WordPress sites at a company in Dallas, TX. Did you know texas has their own power grid - separate from the rest of the USA?

What you've described here would make an AWESOME movie. Granted, I'm not sure how one would work "Datacenter" into the title and still sell many tickets.

What makes it even more exciting is that some of the companies were a bank and a hospital. Lesson learned - a bank gets far more anxious than a hospital when their servers go down.

I'm pretty sure there were Zombies roaming the aisles of servers where the power was out, that might help the box office sales.

Cory Doctorow (of Boing Boing fame) wrote a sci-fi novelette about a datacenter during a crisis: http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked...

I personally didn't find this piece particularly great, but there's an entry in the genre for you.

> multiple 'once in a lifetime' failures that causes it.

That reminds me of hearing mention of "multiple six-sigma events" I think it was, when things were crashing in 2008: the only sane conclusion is that the models are broken.

Reply on news.ycombinator.com