Hacker News Reader

Comment by davidu | Hacker News Reader

Five nines is impossible Really. It’s just not going to happen.

Actually, it is going to happen. One individual part may fail more than 99.999% of the time, but overall system integrity can certainly be designed with greater than 99.999% uptime.

Just get a second datacenter, get a second transit provider. Just as folks scale horizontally, you can built out reliability into the many many 9's such that when one component fails, the overall system integrity isn't impacted.

It's not easy, and not always cheap, but it's quite doable.

View on HN

5 nines is completely unrealistic for the internet. The internet backbone doesn't even have 5 nines. Nobody has 5 nines, not even google.

5 nines is 5 minutes of downtime per year, or an hour per decade. If you have an hour of downtime, you need to have 100% uptime for a decade to achieve 5 nines. The only thing that has 5 nines uptime is the land based telephony network in the US, and only after many decades of effort struggling to get there. Think about the massive amount of redundancy, extra capacity, and backup systems the phone network has, much of it mandated by law. Not merely battery backups everywhere, but backup generators too.

Think about the implications of building a system for actual 5 nines reliability. With today's technology it is very, very costly. If google built their systems the same way the telephone network does they would be operating at a huge loss instead of a hefty profit. Over-engineering is a waste of money. For the vast majority of cases, 5 nines reliability on the web is excessive and unrealistic.

Perhaps "impossible" is overly dramatic, but "economically impractical" may, in this context (general purpose networked computing), be the same thing.

It seems to me that "five nines" is achieved more often with careful use of definitions, rather than careful engineering. Low-hanging fruit are scheduled maintenance and degraded performance.

Otherwise, wouldn't Tandem have been wildly successful?

Complexity costs. It costs money and it costs downtime. Look at Google: they have redundant everything, super-smart people, super good operations and still have some downtime.

How long does it take to fail over to a second data center? Can you do it in less than 26 seconds? I can't.

According to Amazon, EC2 can remove a server from an Elastic Load Balancer rotation at a polling interval of 5 seconds. You can pool servers in different availability zones into the same ELB. I'm not sure the question even makes sense in that context, which I sort of view as the logical extreme of what davidu wrote.

Pardon my ignorance, but what happens when the datacenter hosting the load balancer goes down? Are the load balncers redundant across data centers?

In short, yes. There's a good description of it here. http://clouddevelopertips.blogspot.com/2009/07/elastic-in-el...

I've worked places that had that setup with BGP. Of course, you need to announce the same IP block out of both data centers, your application needs to be able to deal with that, and that sort of thing gets pricy, but it's doable.

Reply on news.ycombinator.com