jespern

You're absolutely right that the cloud is not magic, but you do get some guarantees with EBS. From their website:

"Each storage volume is automatically replicated within the same Availability Zone. This prevents data loss due to failure of any single hardware component."

We don't keep the database on the same EBS, and we have segmented database traffic out to several EBS volumes (for WAL, etc.) That's not the issue.

We take regular snapshot backups. We didn't lose any data. We have everything, we just can't get to it.

Regardless of what might make sense in this situation, it's not working for us. We've moved both our instances and the volumes to different availability zones, to no avail.

I just received a call from AWS engineering, assuring us that we are currently their top priority, and a team of engineers are working to fix the problem. They're seeing the issue on their end, and fortunately for them, it seems rather isolated to our instance.

Could we have taken precautions to prevent this problem? Maybe. We hadn't, cause we didn't anticipate a problem as exotic as this one. The only way to keep persistent data on EC2 is using EBS, and right now, it doesn't work for us, at all. This is not a common problem that could've been solved with backups or snapshots, or whatever.

We've been contacted by several hosting providers, and right now, Rackspace seems pretty nice.

The problem isn't that we don't have failover here, it's that we store all repositories on a single EBS volume. This has worked great for us in the past, but as of last night, that volume has become virtually unavailable to us. It doesn't matter which instance we mount it on, the throughput we get from it is excruciating.

If, or at this point--when, we move, the disk architecture will look different, and general failover will be less of an issue.

Amazon has for the past 8-10 hours been investigating the issue, and we're left pretty dumbfounded as of to what has happened exactly. I'll summarize everything in a blog post once the chaos is over.

I'm here to answer questions if there are any (I run Bitbucket.)

We archived the status page, for those who can no longer see it (since we're back up): http://swag.dk/bitbucket/downtime_21082009.html

Pardon us, we're taking the opportunity to make the site better in the future, for you.

"The Django REST interface makes it easy to offer private and public APIs for existing Django models."

Piston doesn't tie you to a model (although it can), which is very important for any decent sized API. Exposing models is something you want half the time, but there's a lot more to it.

An SPF record would be useful, so mail doesn't get unnecessarily queued with some providers (like GMail.)

I was lucky enough to get Kenichi Yoshida (http://www.kenichiyoshida.jp/). He does excellent stuff, check out his portfolio. Not too expensive either.