What to do when your Amazon EBS fails
lucidchart.com
lucidchart.com
The world is full of wonderful alternatives like SoftLayer, for example (no affiliation). We've been running a very high traffic system there with individual box uptimes of 700 days before we had to reboot them during scheduled OS upgrade/maintenance.
The "AWS scaling stories" I hear at dev. meetups make absolutely no sense. People, AWS original sales pitch was to take care of scaling for you, not the other way around.
The developers obsession with AWS amazes me. It's like paying for a BMW and being proud of carrying around a self-made toolbox in case of an engine failure in the middle of nowhere, and everyone is proud to "know someone inside Amazon". Since when establishing a personal relationship with a car mechanic is seen as "cool"? Why not just pick a quality brand?
You've always got to "waste your very precious time" on something -- putting together servers, dealing with hard drive crashes, etc. No matter what, you have to plan for things breaking, and a lot of the time, they'll break in ways you couldn't plan for.
It's not like there are other cloud platforms out there with zero failures and zero deficiencies. But how, for example, is SoftLayer better than AWS?
Where do I begin... The latency to our dedicated DB servers is always 1-2 milliseconds. Never 5. Never 50. I do not understand how people tolerate 50ms response time from memcached servers on AWS.
We pay much less "per request". Way less. We never have to worry about servers disappearing. We don't need proprietary APIs for server images because the Chef script makes me a DB or a DNS box out of any Linux box.
We do not waste any time "architecting for failure": our databases run on RAID arrays with hot standbys, no need for chaos monkeys. Everything is very, very simple. We never used SoftLayer documentation because nothing of what we use needs to be documented.
And most importantly, we had less than an hour of downtime in 3 years. And none of that was Softlayer's fault.
It becomes much harder to reason about performance, uptime and responsiveness of your system when the underlying infrastructure does not provide guarantees on basic building blocks like I/O latency. Everything on AWS is always in flux and you have to over-allocate resources and/or over-engineer around those faulty building blocks. It may make you feel like a hero to survive in this jungle, but doesn't it look like a "forced premature optimization"?
99% of every graduating YC class could be running on a single pair of dedicated low-end servers and saving themselves a TON of time.
Ha - your in for some fun the first time your hardware fails.
My point is that we didn't have to spend nearly as much money and time as AWS forces you to do.
Another point I want to make: not all hardware is the same. If you're running in Amazon/Google style data center optimized around energy efficiency and real estate density and you're spending serious R&D money on tolerating server failures.
But guess what: you are not Google and you are not Amazon. You need to be in a datacenter which is optimized for individual server longevity. Cooler temperatures. Better hardware. Have a hot stand-by. That's cheap. Such way to "compute" may be expensive for Google scale to run, but it's much cheaper and easier for nearly everyone else.
Chaos Monkey does effectively exactly what you claim to be able to survive - it unplugs servers. The fact that it runs on AWS is inmaterial, it exists to make sure you really can survive a server disappearing. The underlying point is that many teams build their software to survive server failures and never test the fail-over till it occurs for real, that often goes very badly. Chaos Monkey tests fail-over on a regular basis so you find out about it before it costs you hours of downtime.
Sure, the original AWS sales pitch was a bit disingenuous, but it should be obvious to anyone with deployment experience that everything breaks all the time.
It's not about the "quality" (by what metric?) brand IMO - it's about about evaluating a provider, understanding the failure modes, and choosing whether you're going to try to engineer around them, accept some downtime/customer-facing failure, or find a provider with failure modes you like better.
This article is about picking AWS and understanding the EBS failure case, just like how if I picked SoftLayer I'd be trying to understand their network topology and power setup to see where failures might coordinate and try to understand what to do about it.
1) Proven Scale. Netflix. They're bigger than I will be until such time my company can afford to buy it's own data centers. At that point, I can re-evaluate. 2) Operational overhead. Startup, running hundreds of hosts, thousands of requests per second, 5 developers, 0 sys-engineers. 3) Lock in. We use Cloudwatch, Dynamo, RDS, Beanstalk, S3, EMR and more. We also use Heroku. No one else out there has the full suite of offerings. Not using AWS would require us to stitch together offerings from several vendors, some technology wouldn't be feasible having to hop across the internet (do you want the lag to get to your database from one data center to another?) 4) Billing. We get 1 Bill which we can predict. AWS takes credit cards, will do Net 30 billing, and are generally pretty flexible. 5) Customer Service. Don't get me wrong, support at AWS is ... not it's strong suit. We also don't spend enough to be a "big fish". However, if we used 10 different providers to get our infrastructure together, we'd me a minnow to each one. By consolidating, we're a small-medium sized customer which is enough to get into beta's, info on their roadmap ahead of time, and some personal service. 6) Excuses for downtime. So many high profile companies are on AWS that if we have an outage directly attributable to AWS, the odds are a lot of the rest of the internet is down, too. Seriously. It sounds stupid, but it's a huge reason. 2 years ago we were raising money. Right in the middle of our final week (when all our investors were visiting our site and making sure they really wanted to sign on the dotted line), we had an outage that lasted several days. This was the AWS outage that took out Heroku for a few days, and ours was one of the last databases to get restored. If we were using almost anyone else, investors might have been like "oh, this team doesn't really know what they're doing, they're not really that technically competent, I'm not going to invest". But since Netflix, Foursquare, and dozens of other companies were all offline at the same time, the investors were just like "they're using AWS, if Netflix can't keep online with hundreds of engineers, then I'm not going to expect that this little startup is going to do better". 7) Could I really do it better? Is anyone else doing better? AWS is the leader, so we all see the warts ... but at least they keep their certs updated (ahem microsoft)
o Liquidity of engineering talent - thousands of engineers know how to immediately work with the AWS infrastructure
o Design Patterns - They are constantly being updated, discussed, improved upon. People share how to work around AWS issues.
o Commitment to reducing prices - AWS is constantly dropping the prices for everything.
o Commitment to increasing the number of services - AWS is constantly releasing new and useful services on their infrastructure.
o Likelihood of being in business in 3 years. AWS isn't going anywhere. You can rely on them being around.
o Really good post mortems. Each time they fail, they do pretty transparently, and deliver a really painful post mortem that encourages me to believe they are committed to constantly improving.
o Multiple Availability and Region zones. I can, with very little effort, make my services available in redundant regions. I don't think there has ever been a multi-region failure with AWS, and, at most, it's only happened a couple times in their history. Multi-Availability zone failures have happened (and Amazon has apologized - that shouldn't happen) - but, it's < half a dozen times.
o Market Leader Advantage - The market leader tends to win. Microsoft for Desktops, Oracle for Databases, Linux for simple Web/Database/PHP stacks. This isn't always the case (Juniper can beat Cisco in a lot of places) but you know Amazon will be a Top-3 contender for at least the next 5 years.
I realize a lot of these are "Business" reasons, which is incredibly frustrating for people who are comfortable with hosting bare metal, and realize that, given just half the money, they could go into a CoLo, rent a cabinet, buy some servers, routers, firewalls, storage - install it, and return twice the performance that they would get on AWS.
But quite often, the business argument is a more important driver for CxO people making decisions (and assigning budget) than technical reasons.
And, with all that said - I've been in about a dozen Tier-1 data centers in the last 15 years, and, with one exception, every one of them has had multiple sustained outages. (The exception has been Switch SuperNap in Las Vegas - 3 years and no major outage.) - So, failures are part of the business. I'm not expecting Amazon to eliminate them, so it doesn't disappoint me when they happen.
Because those alternatives as just as bad.
>We've been running a very high traffic system there with individual box uptimes of 700 days before we had to reboot them during scheduled OS upgrade/maintenance.
So, just like EC2?
As far as I know, it was never about that, but about giving you the tools to spin-up whatever you need at will. The cool thing about AWS has always been the APIs.
I would rather take a service that has downtime but give me the tools to failover/recover than a service that promises something they cannot promise (100% uptime) and when it inevitably fails, I'm at the mercy of their support.
People who need instant scaling have limited alternatives, but not sure why everyone is sticking with AWS.
> The process was stuck in the kernel waiting for an IOCTL, so killing the process did nothing – even kill -9. It was going to stay that way until EBS was back to normal.
I did not know that can happen! Can anyone provide a further explanation/material for why this is the case?
https://jira.mongodb.org/browse/SERVER-6178
http://www.noah.org/wiki/Kill_-9_does_not_work
http://stackoverflow.com/questions/223644/what-is-an-uninter...
The first is the bug I reported to 10gen about their MongoDB setup. Contained in the bug report is the set of commands I used to reproduce the issue so that you can see it yourself.
The second and third links are about processes in uninterruptible sleep, which is the state my MongoDB processes were in. It should give more detail than I'm about to give.
Linux processes all have a state (running, sleeping, etc). One of those states is "Uninterruptible Sleep". Only the kernel can change a process's state. Uninterruptible sleep ("D" in `top`) is a state in which the process will receive no interrupts (signals). Since the `kill` command is just a glorified signal sender, it ceases to work for those processes.
It's possible to wait in a "interruptible" way, and many subsystems do. But that's hard, because it means that the resources you would otherwise be responsible for clean up need to be detected and cleaned up by someone else (or alternatively that someone else needs to pick up and finish what you started, or you need to be able to roll back a partially completed action, etc...). Userspace usually ignoes this problem (because the task got killed, right?), but in the kernel a leak is a huge bug.
So the default is that waiting in the kernel means that the kernel can't allow the process to be terminated (or otherwise be delivered a signal that would invalidate the stack). And the side effect is that userspace sees an unkillable process in the "D" state.
The other great thing about corosync+pacemaker is that it's super easy to write your own HA scripts. Need mongodb? Then write your own script(not sure why you'd use corosync for this, but you could...).