Hacker News Reader

Comment by chintan | Hacker News Reader

We learnt our lesson the hard way after the great AWScalypse of Apr 2011.

The lesson: Use n>1 hosting companies (even if one of them promises a-z-multiregion-distributed-fault-tolerant-back-up)

My co-location provider's data center has had about four hours of downtime in ten years. It turns out a backup diesel generator, redundant connectivity, and a good network/sysadmin are all anyone needs.

People are now working out how to failover from AWS to Rackspace and that is infuriating to me. You... you need redundant clouds now? That can't be right!

You're comparing different things: most of the complaints about AWS have been due to servers (EC2) or network storage (EBS) failing. The outage in March was networking related so that would count, however.

Why do you assume amazon doesn't have " a backup diesel generator, redundant connectivity, and a good network/sysadmin"? And if they do have that why is your situation different?

You tell me man. I guess the difference is I can install many more times more server horsepower for the $500/month rental of a half rack at my colo, and the result will be better uptime than EC2. I just can't install them immediately.

how are you quickly failing over from one to another?

We work with 3 providers. Today 2 of then go down. Just Linode is ok now.

ironically, we moved away from linode last month because they kept doing maintenance and turning off our machines without telling us

really? I've noticed an increase in maintenance with Linode recently but they've always been really great with notice, so much so I am consistently surprised. If there is ever going to be planned downtime I've got an email at least a week in advance and it gives me the option to migrate my linode to another server at my own convenience if their schedule isn't compatible with my needs. Do you not get these emails and options?

For the scheduled ones, I think I did receive some prior notice, I might have gotten particularly unlucky with a bunch of emergency/network/etc. maintenance affecting many of my servers simultaneously.

Really?

I had 2 (short, scheduled) downtimes with Linode. Since 2009

Interesting. We moved away from VPS.net for a similar reason. There were multiple occasions where hosts got shut down and support couldn't tell us the reason for it. Since we're using hand-rolled virtualization on top of a few rented servers, we're basically problem-free.

Linode sends out notices for every single maintenance event, which includes both emergency and scheduled events. The maintenance you experienced last month is explained here: http://blog.linode.com/2012/06/13/xen-security-advisories-an...

My problem with them is they don't provide RFOs (Request for outage) because they can't give away any of the "propitiatory secretes" about their setup. I'd like to know what happened when it effects me. Also when it comes to replacing hardware or preforming maintenance, their priorities wont always line up with mine.

If your site has little dependency among itself, then you can use DNS round robin to include IP's from all hosting companies.

If you use DNS round robin, then you pretty much guarantee that everyone will be affected if just one hosting company is down. DNS round robin is not the tool for this job.

Reply on news.ycombinator.com