Designing AWS Architecture to Withstand Outages | Hacker News Reader

Designing AWS Architecture to Withstand Outages

madeiracloud.com

38 pointsdanoprey17 comments

We've seen floods of these "What you should have done if your application was interrupted Friday night" articles, but a common theme is to leave out the fact that many of these sites require synchronized data storage, which Amazon doesn't support between regions. If your organization is shelling out the money for something like Riak, Oracle, or the mess necessary to support such an architecture though the likes of MySQL or Postgres, this is all well and good (save for the data transfer overhead costs).

If your application is architected to use SimpleDB, DynamoDB, SQS, or even RDS, these simple "You should have been using Route 53 and multiple regions" articles get increasingly frustrating. Most applications simply aren't elementary enough to fit into that boilerplate structure, and getting around that fact either requires switching away from Amazon's managed database services (lots of money) or writing synchronizing scripts that play nicely with your stack and launching them on additional servers (lots of money and lots of resources).

While ideally I'd like to see Amazon release some sort of feature for multi-region sync, it would be interesting to see how the tried-and-true multi-region businesses have approached this problem.

Completely true, there's a lot to be written on the subject and there are some great detailed posts out there.

I think Google Compute Engine does exactly that, with a direct connection between regions so the entire system works as one network. I'm not sure if that will be the case outside the US, though.

I don't know if I'd trust Amazon to run links between their data centers. They had a couple routing related outages just this year alone (which lasted for some time), and adding complexity would make that worse. You could easily lose interconnections between hosts in AZs in distinct 3 regions, partitioning everything, making it irrelevant that all 3 actually stay up and are accessible to outside users.

I looked at Azure to see how these things work there. Here is my current understanding: blob and table storage in Azure Storage is automatically geo-replicated between regions. If you don't want that you can reduce your storage cost by about 30%. VM disks are stored in Azure blobs, so if a region goes down you should actually be able to create a new VM in a different region and attach the SAME disc you had attached to the VM in the region that went down.

For the SQL datbases they provide a sync service that can replicate your database automatically between different regions (or even a local database).

I have no clue how reliable all of this is, but at least in theory it looks as if it might be quite a bit easier to do full geo replication across different regions on Azure.

Anyone with actual experience?

I wish I could upvote this 50 times.

The diagrams show two separate masters in different regions with their own slaves, yet DNS is essentially randomly delivering users to each "half". There is no mention of how this is handled, or even that you would have to consider it.

Agreed, every step in this could be an article in itself, but the intention was just to give an overview of all the options.

The only problem I have with this, is that for a really simple system consisting of 3 components (web+app+db) now I need to deploy (and pay for!!) 24 components.

Yes I know, business continuity and stuff. Still, just doesn't feel right somehow.

Most businesses can make the conscious decision to go with the simple 3-component option and accept that they might only get three or four nines of uptime, depending on how flaky AWS decides to be that year.

If you run a service where the worst case scenario for your site being down for an hour on a Thursday afternoon is "millions of dollars are lost" or "our high profile customers go out of business in a way that's directly traceable back to us" then yes, you need all 24 of those things.

If, on the other hand, the worst case scenario for your site being down for an hour on a Thursday afternoon is "some of our customers have to manually post to Facebook so that their friends know that they've been for a run", you can probably shave about 21 nodes off that diagram.

Well said! Often the right thing to do in an outage is "wait it out".

Exactly. You have to weigh cost of the worst case downtime vs cost of redundancy (and not to mention the added complexity of keeping all redundant systems in sync)

You have options. You can pay for and run just three components on AWS, and still have the power of dynamically scaling your application when you need to.

You can then pay for three really cheap VPS's and load balancers around the world using something like Linode (or a much less reliable VPS, doesn't really matter) and replicate your data occasionally to the VPS. When your AWS instance goes down (because you foolishly rely on the same Virginia datacenter that has more breakdowns than Lindsay Lohan) you cut over to the hot spare VPS and rate-limit your incoming requests until AWS comes back.

You end up paying for 6(ish) services, still relying mostly upon AWS but with a tiny DR site you can use during emergencies.

You can get away with just a db replica to another region. In the event of a major outage in the primary region, promote the replica and boot the other stuff you need in the secondary region.

Don't bother with AZs, they've proven not to be an independent unit of availability.

Not to mention it will cost a lot of money.

Off topic but their product, madeiracloud.com, looks very slick.

Does anyone have any experience with it?

Have a try, no credit card or AWS credentials required ;-)

We're still in early days but we did a Show HN a little while ago: http://news.ycombinator.com/item?id=3808031

Interesting, but not at all new. A really interesting article on the subject would be how to do it while having nearly consistent data across the data centers. In the case of the proposed approach, you have simply no communication between the DCs.

Hi Loic. I agree, only meant to serve as an introduction to the subject for people who are not familiar with it, which seems to be way too many people.

Reply on news.ycombinator.com