Comment by dctoedt | Hacker News Reader

Comment by dctoedt | Hacker News Reader

I'm baffled that the powers that be in NYC didn't learn from Houston's experience with Tropical Storm Allison in June 2001. All the hospitals at the Texas Medical Center had their basements and ground floors flooded; there went their generators [1]. (I was part of a large group of volunteers from all over the city that helped to evacuate patients for transfer; it was spooky seeing a grand piano floating in a below-grade-level lobby of one of the hospitals.)

My then-employer's building had its basement flooded; we were on floors 19 through 25, but our electricity and phones and Internet were gone --- and this, three weeks before the end of Q2. Our developers and IT people hauled computers down the stairs, and we moved to temporary space for several weeks, but we still missed, that is, we didn't achieve the sales and revenue targets that we had forecast for analysts.

I'm given to understand that because of the lessons learned from Allison, Houston isn't quite as vulnerable to flooding any more.

[1] http://en.wikipedia.org/wiki/Tropical_Storm_Allison#Texas

The bottom line is that people just don't plan for 100 year storms. I suspect that's because the number of things that go wrong once every 100 years is so large and the cost of handling each one is so disproportionate to the risk that it's not cost effective. Think about it this way: under what circumstances would you double your hardware budget to avoid 1 day of downtime every 100 years?

When the Brooklyn-Battery tunnel is flooded, it's hard to blame anyone else in the city for being insufficiently risk-adverse.

You might as well blame a data-center for going down when the building it's in burns to the ground.

This is exactly right. The truth is you can spend infinite money on various scenarios, and that cost has to be paid. The manufacturer side of this is 'warranty', you guarantee that something won't break in 90 days, a year, 5 years, what have you and you price to that probability.

Another risk in New York that will take data centers offline, earthquake. It will happen at some point, but the chance of it happening in the next 100 years small (contrasted to Santa Clara where we're much more likely to get an earthquake than a flood). So you could pay to have your Manhattan high rise put onto base isolators and proofed against an earthquake, but what size earthquake? Magnitude 3? Magnitude 6? Magnitude 9? And then if your building is still standing brightly after the Magnitude 9 earthquake are your fiber optics still there? How about the network tie point? Did it fall into a hole in the ground? So the cost to make all of Manhattan resistant to a 9.0 earthquake?

Its impressive that they are carrying up the fuel. I might be inclined to see if I could tractor in a 12kW generator to run one elevator. Sure you'd be burning fuel at both ends but it would be easier on the crew hauling the petrol.

due to flooding in the basement and diesel fuel spilled everywhere, they wouldn't power up the elevators even if they could. The building's entire electrical system is under salt water/diesel mixture.

Your team's dedication is amazing.

Having been lucky enough not to be placed in the situation you are in, I doubt I could have gone to such lengths to get everything back online so quickly. I admire everything you guys are doing.

Thanks.

Magnitude 5.0 is 1 in 100. Magnitude 6.0 is more like 1 in 500. You do a cost-benefit analysis for the different return rates.

Hospitals should definitely be built to handle a magnitude 5 or more. You're talking about a high probability of loss of life if they can't handle a 1 in 100 year event. Data centers, who cares? It's cost vs expected damage, and unless it's a safety-critical server (emergency services co-ordination?) you're probably just as well off shipping the servers somewhere else and eating the cost of the downtime.

As for getting a 12kW generator in a natural disaster, generators are in extremely high demand. You'll have trouble finding one. The other fun thing about generators in a disaster - people usually don't test them, and end up panicing in a dark basement with no idea how to switch them on and hook them up.

Reply on news.ycombinator.com