> If your app is down, it's only because you don't care about the availability of your service.
That is absolutely absurd. At what point did the common-sense solution to "unacceptable downtime on AWS" become "buy two of everything"?
That is absolutely absurd. At what point did the common-sense solution to "unacceptable downtime on AWS" become "buy two of everything"?
We operate systems that sit on the pages of the top e-commerce companies in the world. We have 10 separate segments of clusters. Operating in four AZ's in East, three AZ's in West-1 and two AZ's in West-2. When this outage happened, the servers that were impacted in east were removed from our DNS and within 9 minutes the impact of this event on their sites was eliminated.
At Quantcast we have physical servers in 14 cities. We use anycast to achieve site failovers in 6 seconds. Downtime for us would impact millions of websites, so we don't have downtime.
(The trend for regular ISPs is probably improving, except that mobile/carrier DNS is often particularly broken. It would be interesting to do monthly surveys of this.)
It's nontrivial to determine exactly when to drop the announcement. And be careful, because if you are too eager to drop the announcement, you may do it in more than one site at a time.
At first we used DNS with short timeouts, but those timeouts are only advisory and are ignored by some implementations. We would see most traffic tail off within 10 minutes for a one minute timeout, but it took many hours for all the traffic to migrate over to the new DNS. The folklore on using less than one minute for a DNS timeout is that a huge percentage of implementations ignore sub-minute timeout. Funny how much of the Internet's operation is passed along as folklore and not really known for sure.
Thanks for asking. Hacker News should be about sharing best practices and making the Internet a more reliable place.
The only useful/sane use case I can see in Amazon EC2 would be for services like Heroku where they need to automatically be able to manage a truckload of VM's as their rapidly growing infrastructure, unless you want to do it yourself which I imagine is quite a headache unless you work closely with someone like Amazon or Rackspace.
Yes white boxes are cheap. Site negotiations, design, procurement, networking, operations, and maintenance are expensive in dollars and time. Personally I run "a bunch" of physical sites across the globe. It would be waaaay easier to be able to turn up rackspace/aws/google instances as needed.
How do cloud services compare to a gym membership? Are you implying you can't get out of your AWS contract?
You'd be surprised how many people that actually use EC2 think it is.
> Yes white boxes are cheap. Site negotiations, design, procurement, networking, operations, and maintenance are expensive in dollars and time.
It's called planning ahead of time. If not, then here's a suggestion: Use EC2 until you set it up and migrate, if you cannot wait that is.
All in all I don't mind whether people use EC2 for whatever reason. Just stating my opinion. I agree of course that in terms of "convenience" is has the upper hand. Not having to wait for boxes to be added to data centers, being able to spin up boxes in multiple regions through a single company/console. Maybe your use case does justify using EC2. Many other people clearly do not (hence all the whining because of all the downtime, which they wouldn't have had if they deployed to multiple AZs/Regions).