Hacker News Reader

Given a choice between hosting servers on AWS, and trying to build my own reliable infrastructure with a single sysadmin, I'll take AWS in a heartbeat. But I do want to quibble with one of your points:

It is easy to be an arm chair coach and scream that... they should have predicted this storm

I'm not as smart as the AWS developers, and I have a lot less experience with large-scale distributed systems.

But thanks to my own cluelessness, I've blown up smaller distributed systems, and I've learned one important lesson: Almost nobody is smart enough to understand automatic error-recovery code. Features like automated volume remirroring or multi-AZ failover increase the load on an already stressed system, and they often cause this kind of "storm."

So I've learned to distrust intelligence in these matters. If you want to understand how your system reacts when things start going wrong, you have to find a way to simulate (or cause) large-scale failures:

This is something that Google does really really well by the way, I've watched them turn of 25 core routers simultaneously carrying hundreds of gigabits worth of data, just to verify that what they think will happen, does happen. http://news.ycombinator.com/item?id=2475112

You also need to pay particular attention to components with substantial, ongoing problems, and make sure you don't let known issues linger:

I work at Amazon EC2 and I can tell you what's going on (thanks to this handy throwaway account). What's happening is the EBS team gets inundated with support tickets due to their half-assed product. Here's the hilarious part: whenever we've asked them why they don't fix the main issue, they keep telling us that they're too busy with tickets. What they don't seem to realize is that if they fixed the core issue the tickets would go away. http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...

Now, I'm not saying I could have done any better than Amazon (evidence suggests otherwise). But I do know that I'm not smart enough to understand these systems without testing them to destruction, and aggressively fixing the root causes of known problems.

I would be surprised if Amazon didn't do testing similar to the Google quote above. AWS is insanely complicated, and the problem with an insanely complicated system is there are infinite failure modes (or such a big number it is close enough to infinite for human purposes). As such, it is impossible to test all of them.

But thanks to my own cluelessness, I've blown up smaller distributed systems, and I've learned one important lesson: Almost nobody is smart enough to understand automatic error-recovery code. Features like automated volume remirroring or multi-AZ failover increase the load on an already stressed system, and they often cause this kind of "storm."

It's basically Test-Driven Development: if you cannot test it, don't write it.

It is hard to test emergent behavior in large distributed systems, you pretty much have to actually run the tests live to see what is going to happen and see if it aligns with your predictions.

No one seems smart enough though ;) http://groups.google.com/group/google-appengine-downtime-not...

> There is no way you could do better, and it is unlikely that if you are building anything that needs more than a handful of servers you could build anything more reliable.

I can't disagree, but there is one key benefit to not using the cloud for some services.

When your company is working on an important deadline, your sysadmins could choose not to implement that pending network configuration change during that crucial period. You can control your own at-risk times, which you can't generally do with IaaS.

As with everything, it's a trade off.

See, this makes me uneasy. The best and brightest work very hard to avoid memory errors in their C codebases then, when errors occur anyway, console themselves that no-one could have done any better. The less brilliant look into more automatic forms of memory management. AT&T hired the best and brightest to make heroic efforts so that the old circuit-switched Bell system could achieve mediocre reliability. I certainly hope that the best and brightest at AWS aren't intending to tackle their enormous control-backplane SPOF by assiduously patching every bug that turns up in its behaviour.

If it's too hard for the best then the concept is dead. But I don't believe it. They made some avoidable mistakes.

Just look at the pattern emerging from these kinds of incidents. There's an automatic cluster recovery mechanism that works for individual node failures but makes matters worse once a larger number of nodes fail.

I wonder whether they did extensive testing or simulation of that scenario. The initial root cause is probably unpredictable because there may be many, but what follows is not unpredictable.

I'm not ready to concede that because they are such an insanely smart elite group of people we just have to live with week long outages.

I think it's more of a learning experience.

In the grand scheme of things, it's still day one for computing services like AWS.

The problem with all this is that you never hear about the many many failures that occur (daily, hourly at the scale they operate?) that don't cause an outage.

I think you're drawing the wrong conclusions from the grandparent post. It's not These guys are the best, so we will just have to live with these failures. It's These guys are smart people sailing in uncharted waters, so they're going to get a little lost now and then. The second conclusion implies that these waters will eventually become charted, so everyone will be able to avoid these problems in the future.

To me, the parent as well as your comment sounds a little bit too apologetic. The waters are not totally uncharted and the outage was disproportionate. AWS isn't a research project, it's a commercial offering, so they have to take some blame. That re-mirroring storm was not completely unforeseeable. It's exactly the thing you have to consider when designing these kinds of systems.

I've been working at Google, and had a similar experience. But I do think in many cases you can actually do better. It's not that you're smarter than Amazon/Google/... engineers, it's that you're solving a problem that it's orders of magnitude easier.

Your start-up is probably not dealing with setting up & maintaining 200K servers as Amazon, so up to a limit you can actually be better going on your own.

Yep, that was my point with the 'handful of servers' line.

But even that is pretty darn hard. You still have to deal with split brain situations, do all the right things in those cases etc.. If you are a tiny startup that isn't going to hire a dedicated sysadmin (very few of which can build that type of thing without buying some expensive hardware) then EC2 is probably a better choice.

Even just the hardware costs make it pretty braindead. We just shut down the dedicated three system setup I built because it just didn't make sense financially, and that despite me spending a month learning the intricacies of heartbeat it still had odd failure scenarios that we don't experience with EC2. Again, I'm just not that smart. :)

Agreed with the expertise thing, but to add to your summary of the post-mortem, it looks like human error compounded by: 1) An architecture bug (the EBS "control plane" cuts across Availability Zones and EBS clusters, leading to a single point of failure: this is what broke the "service contract"), 2) a spec/programming bug: No aggressive back-off on retry attempts of EBS ops, and 3) two separate logical bugs: the race condition in the EBS nodes & problems with MySQL replication.

I think that's everything. It just goes to show, most disasters in very well-engineered systems are generally the result of a series of things all going wrong at once, not individual failures...

Agree completely. We are still in the early days of AWS-like technology. Providers and users both will experience some serious issues in the years to come.

Heck, users of municipal water systems still experience outages and that technology is arguably mature.

The really encouraging thing is that the Amazon post-mortem writeup indicates they're taking it very seriously.

My tl;dr reads differently: "we lacked appropriate congestion control in our EBS recovery algorithms."

The trigger was a bad and unexpected network configuration change, but the error was that the attempted recovery by the stuck volumes was that it was uncontrolled.

I don't think that anyone is knocking the intelligence of the AWS engineers, or saying that anyone else could do it better. Just like NASA engineers and scientists are incredibly intelligent and good at what they do, systems can become complicated enough that unexpected errors creep into the system, and not any particular component.

The best are human beings too! I mean even the best work in two ways. Build on best principles already known, and make your own best principles. The first depends on how much you know, how much time you have taken to learn, implement, practice and perfect. The second depends on how far you can see.

In either case you can always overlook or fail to predict the even the easily foreseeable future. And that happens due to many reasons plain human error, or even over confidence which is some times the case with the best.

The best way out of this problem is what Jeff Atwood had blogged some time days back. Is to keep failing, and keep failing in different ways. And each failure needs to be translated into to lessons of some sort and then the solution of it a best practice. Even the Netflix model of failing purposefully will do.

There is no way the best can be flawless. Nothing is flawless, so as long it is done by a human.

Agreed. It has actually given a very nice excuse to developers like me. Whenever your manager comes up to find why something that you are working on is not up, you can simply reply with something like this -

If AWS with some of the smartest engineers can be down for that long do you think that our crappy service will be up 100% of time?

A single manual configuration mistake should never be able cause this type of complete system failure. For AWS to be engineered this way, the engineers you met must not have been as smart as they seemed. EBS may be extremely sophisticated in other respects, but if this type of thing was even a possibility, the AWS team are very fallible, and probably far from the best in their field.

>In short, take my word for it, the people working on this are smarter than you and me, by an order of magnitude. There is no way you could do better, and it is unlikely that if you are building anything that needs more than a handful of servers you could build anything more reliable.

Ever since the AWS outage, I've seen a number of these "the AWS guys are so smart, I've met them." type comments. And then paraphrased: "There sort-of can't be that much to blame on them because of how smart they are, and they are so smart anyway, who could do better?" That's not a valid argument, not everyone is equally impressed by an individuals intelligence, perhaps your assessment is wrong. And even if someone is insanely smart, they still can commit practical errors which indicates they are smart, but still flawed in their understanding of engineering in significant ways. Perhaps AWS simply does need a higher caliber of engineer that wouldn't miss out on these dead-simple safeguards that would have prevented this outage.

It's not so much about thinking that somebody else could do better.

By doing what they do they create the expectation that they _are_ doing better than everybody else.

And let's face it, despite bugs and human error that should be avoidable, they are.

At that scale, MAYBE.

In the last two years my workplace has gotten pretty good at handling SAN outages (due to terrible Oracle equipment).

Put simply, this set of scenarios can't happen at my work-place. We don't have that level of automation, there's only a pair of SAN systems in the mirrors, there's no "hunting for capacity".

I'd suggest most businesses are closer to this than AWS.

There's a lot of sugar coating going around saying "You couldn't build a better space-shuttle", and that's probably true. But if I only need an extremely reliable bicycle that's a false argument. The Simplest Thing That Could Possibly Work doesn't apply just to programming.