Hacker News Reader

Comment by spolsky | Hacker News Reader

The bottom line is that people just don't plan for 100 year storms. I suspect that's because the number of things that go wrong once every 100 years is so large and the cost of handling each one is so disproportionate to the risk that it's not cost effective. Think about it this way: under what circumstances would you double your hardware budget to avoid 1 day of downtime every 100 years?

View on HN

This is exactly right. The truth is you can spend infinite money on various scenarios, and that cost has to be paid. The manufacturer side of this is 'warranty', you guarantee that something won't break in 90 days, a year, 5 years, what have you and you price to that probability.

Another risk in New York that will take data centers offline, earthquake. It will happen at some point, but the chance of it happening in the next 100 years small (contrasted to Santa Clara where we're much more likely to get an earthquake than a flood). So you could pay to have your Manhattan high rise put onto base isolators and proofed against an earthquake, but what size earthquake? Magnitude 3? Magnitude 6? Magnitude 9? And then if your building is still standing brightly after the Magnitude 9 earthquake are your fiber optics still there? How about the network tie point? Did it fall into a hole in the ground? So the cost to make all of Manhattan resistant to a 9.0 earthquake?

Its impressive that they are carrying up the fuel. I might be inclined to see if I could tractor in a 12kW generator to run one elevator. Sure you'd be burning fuel at both ends but it would be easier on the crew hauling the petrol.

Magnitude 5.0 is 1 in 100. Magnitude 6.0 is more like 1 in 500. You do a cost-benefit analysis for the different return rates.

Hospitals should definitely be built to handle a magnitude 5 or more. You're talking about a high probability of loss of life if they can't handle a 1 in 100 year event. Data centers, who cares? It's cost vs expected damage, and unless it's a safety-critical server (emergency services co-ordination?) you're probably just as well off shipping the servers somewhere else and eating the cost of the downtime.

As for getting a 12kW generator in a natural disaster, generators are in extremely high demand. You'll have trouble finding one. The other fun thing about generators in a disaster - people usually don't test them, and end up panicing in a dark basement with no idea how to switch them on and hook them up.

due to flooding in the basement and diesel fuel spilled everywhere, they wouldn't power up the elevators even if they could. The building's entire electrical system is under salt water/diesel mixture.

Your team's dedication is amazing.

Having been lucky enough not to be placed in the situation you are in, I doubt I could have gone to such lengths to get everything back online so quickly. I admire everything you guys are doing.

Thanks.

When the Brooklyn-Battery tunnel is flooded, it's hard to blame anyone else in the city for being insufficiently risk-adverse.

You might as well blame a data-center for going down when the building it's in burns to the ground.

Reply on news.ycombinator.com