AWS Server Issues Take Down Instagram, Vine, Airbnb And IFTTT | Hacker News Reader

AWS Server Issues Take Down Instagram, Vine, Airbnb And IFTTT

techcrunch.com

How many times do we have to hear about EBS issues and companies like Reddit reporting EBS being a disaster before we stop seeing outages from it? Either, Amazon is looking into it, publicly, or people have wised up to the facts and stop using EBS-backed services. Those include Elastic Load Balancer (ELB), Relational Database Service (RDS), Elastic Beanstalk, and others. It's just part of the AWS fabric; you "need" to be in multi-region, multi-AZ, and architect to be on ephemeral disks that can disappear at at any instant.

I agree with you 100%. For those of us needing to run MySQL, what would be the best ways to minimize the dependency on EBS and maximize HA?

I haven't implemented this on EC2 myself, but Galera Cluster is a godsend for MariaDB/MySQL HA. Multi-AZ is probably viable, but multi-region may hurt.

A multi-AZ cluster with ordinary binary log replication to another region (with preparations to launch a new cluster in that region based on that slave on short notice) might be a good solution.

You would not want to implement PXC across regions - the round trip time would be too hard in individual transactions, and the ISTs would be murder to your bandwidth. Between AZs should be better, but remember with PXC that your transaction time is limited by your network round trip to the slowest node.

Regular asynchronous replication is your best bet, with a regular run of pt-table-checksum scheduled to ensure your data is consistent.

For those confused (as I was at first), falcolas is referencing Percona XtraDB Cluster, a particular distribution of MySQL + XtraDB + Galera Cluster.

> Regular asynchronous replication is your best bet

That doesn't get you HA in any practical form. I've done the heartbeat thing, it created more outages by itself than would have occurred with no HA solution at all, usually failed to kick in during a real failure, and we could never fix all the split-brain scenarios. Galera is absolute magic by comparison.

Hence my suggestion that a cluster be deployed within one datacenter, with asynchronous replication to a standby.

Galera is magic, I'll agree, but there are just too many shortcomings for me to recommend it for most people.

Heartbeat is very problematic, I agree, but it's hardly the only solution out there (and far from the best solution). That said, I have learned that it's remarkably configurable, so many of the problems you encountered could probably be addressed, if you're willing to learn about Pacemaker and really dig into its configuration.

I wish you'd elaborate on those shortcomings, because individual transaction latency is the only real one I've found, and for most workloads, it's not nearly enough to overcome the HA and CPU scaling benefits.

We run a master-master setup between Oregon and Virginia, and its served us very well. We monitor MySQL instances for multiple customers, so uptime is a must; having a multi region setup has filled that role perfectly.

Can't you use RDS? It has Multi-AZ replication and failover. It may seem costly, but it means you don't need to manage all that stuff yourself.

I should mention that the system I'm building that uses it has yet to go into production, so others may have more experience on this than I do.

We're already using Multi-AZ, but still had DB connectivity issues throughout the outage.

EBS has its problems, but I think a blanket rejection of EBS is probably unwarranted. It's true that in 2011, EBS was very shaky. Amazon has since invested quite a bit of energy into EBS; at that time it was a new service that couldn't handle its own growth. EBS has become substantially more reliable, both in terms of availability and performance consistency (provisioned IOPS).

That doesn't mean you should rely on it absolutely, at least within a single AZ. Multi-AZ is a critical part of any AWS deployment at scale, if not multi-region (depending on your needs). EBS is not a cure-all, either; if you have higher performance demands than it offers (between provisioned IOPS and IO-optimized instances) then that's fine, just measure first.

Finally, I'd like to point out that Reddit's complaints about EBS aren't the best to echo as an example of EBS issues: first, they are a few years old, and at the time both Reddit and EBS were architected with substantial deficiencies.

Are Instagram, Vine and Airbnb NOT multi-AZ??

I can't say for sure, but they could be relying on an EBS-backed service (ELB, RDS, etc) or suffered from an AWS issue that suffers even in multi-AZ. A lot of AWS outages could have only been mitigated with multi-region.

Amazon claims the issues only affected 1 AZ.

From approximately 12:51 PM PDT to 1:42 PM PDT network packet loss caused elevated EBS-related API error rates in a single AZ, a small number of EBS volumes in that AZ to experience degraded performance, and a small number of EC2 instances to become unreachable due to packet loss in a single AZ in the US-EAST-1 Region. http://status.aws.amazon.com

So.. are we to believe that Instagram, Vine, etc. are not properly architected for multi-AZ failover? Or is there more to this than what Amazon is reporting?

Bottom line, why did these apps have any significant downtime at all?

Potentially, they do not have multi-AZ ELBs and only thought to have their EC2 or similar instances in multi-AZ. Or they don't have proper tuning of their DNS to stop routing to failing ELBs. Or... Amazon is unaware or not making us aware of the side-effects that their subnet outages tend to have on the overall network in a region.

EBS is now per-zone. So as long as you are properly distributed across zones you should be fine.

The only caveat to this would be if you have a single point of failure. Like a PostgreSQL database or something else that just doesn't have a good high availability story.

Interesting that Instagram is still running on Amazon - I wonder when they'll be hosted off of FB's in house servers.

Amazon claims[1] the problem was isolated to a single AZ. Yet it took down Instagram, Vine, Airbnb and more.

Simple question: Are none of these major apps properly architected to failover to other AZs, or is Amazon lying?

[1] http://status.aws.amazon.com

Well, I spent the afternoon performance-testing EBS-backed instances in us-west-2. Performance was no worse than usual. So I credit the idea that the problems were limited to us-east-1, and the companies mentioned should do more work on cross-AZ resiliency.

us-east-1 is a region, which has multiple Availability Zones.

We run many (hundreds) of instances in most/all AZs, and this appeared to be about 30 minutes of flakiness in EBS instances created during 2012 (roughly) in only one us-east-1 AZ (Amazon randomizes the names, so my us-east-1e is different than yours).

Data point in favor of Amazon.

This is where things like Chaos Monkey really come in handy. If you want to run EBS or ELB, I suggest randomly breaking things, until your architecture is resilient.

This is one reason I like Linode. They don't offer EBS, so you don't design it into your system. Also, they are just a really great company to do business with.

The paranoid part of me sees an untimely coincidence between this and the recent outages at Google and Amazon.

All your EC2 instances are belong to NSA?

Are you saying the Amazon outages may be related to the Amazon outages?

These involve subsidiary hosted services. I was referring to the online storefront itself. Sorry for not making the clarification! :)

Interesting, when you're a quite successful company - why should you be critically dependent on such 3rd party services?

I mean, why not have an in-house hosting with the appropriate staff? Is this so much trouble, even for rich companies?

Another side is having a several in-house servers in different locations.

Because when you start designing the architecture of your in-house hosting to be as flexible and reliable as possible, you end up with something very similar to AWS, only with a fraction of the resources and experience to manage it.

It's not like you suddenly stop having technical issues because it's your own staff and hardware.

When it comes to your company, you are aware of more details and plans, so I believe you can get off with the less flexible but more tailored system.

You won't suddenly stop having technical issues, but at least you'll have a much more control.

Thanks AOL for letting us know that AWS issues affect lots of sites!

Reply on news.ycombinator.com