Hacker News Reader

Comment by old-gregg | Hacker News Reader

Can I ask: why do people chose to waste their very precious time writing software to deal with EBS failures (and other AWS deficiencies) instead of just not using AWS?

The world is full of wonderful alternatives like SoftLayer, for example (no affiliation). We've been running a very high traffic system there with individual box uptimes of 700 days before we had to reboot them during scheduled OS upgrade/maintenance.

The "AWS scaling stories" I hear at dev. meetups make absolutely no sense. People, AWS original sales pitch was to take care of scaling for you, not the other way around.

The developers obsession with AWS amazes me. It's like paying for a BMW and being proud of carrying around a self-made toolbox in case of an engine failure in the middle of nowhere, and everyone is proud to "know someone inside Amazon". Since when establishing a personal relationship with a car mechanic is seen as "cool"? Why not just pick a quality brand?

View on HN

I know several systems administrators who have spent a lot of sleepless nights dealing with SoftLayer networking outages over the last few months.

Sure, the original AWS sales pitch was a bit disingenuous, but it should be obvious to anyone with deployment experience that everything breaks all the time.

It's not about the "quality" (by what metric?) brand IMO - it's about about evaluating a provider, understanding the failure modes, and choosing whether you're going to try to engineer around them, accept some downtime/customer-facing failure, or find a provider with failure modes you like better.

This article is about picking AWS and understanding the EBS failure case, just like how if I picked SoftLayer I'd be trying to understand their network topology and power setup to see where failures might coordinate and try to understand what to do about it.

>> why do people chose to waste their very precious time writing software to deal with EBS failures (and other AWS deficiencies) instead of just not using AWS?

1) Proven Scale. Netflix. They're bigger than I will be until such time my company can afford to buy it's own data centers. At that point, I can re-evaluate. 2) Operational overhead. Startup, running hundreds of hosts, thousands of requests per second, 5 developers, 0 sys-engineers. 3) Lock in. We use Cloudwatch, Dynamo, RDS, Beanstalk, S3, EMR and more. We also use Heroku. No one else out there has the full suite of offerings. Not using AWS would require us to stitch together offerings from several vendors, some technology wouldn't be feasible having to hop across the internet (do you want the lag to get to your database from one data center to another?) 4) Billing. We get 1 Bill which we can predict. AWS takes credit cards, will do Net 30 billing, and are generally pretty flexible. 5) Customer Service. Don't get me wrong, support at AWS is ... not it's strong suit. We also don't spend enough to be a "big fish". However, if we used 10 different providers to get our infrastructure together, we'd me a minnow to each one. By consolidating, we're a small-medium sized customer which is enough to get into beta's, info on their roadmap ahead of time, and some personal service. 6) Excuses for downtime. So many high profile companies are on AWS that if we have an outage directly attributable to AWS, the odds are a lot of the rest of the internet is down, too. Seriously. It sounds stupid, but it's a huge reason. 2 years ago we were raising money. Right in the middle of our final week (when all our investors were visiting our site and making sure they really wanted to sign on the dotted line), we had an outage that lasted several days. This was the AWS outage that took out Heroku for a few days, and ours was one of the last databases to get restored. If we were using almost anyone else, investors might have been like "oh, this team doesn't really know what they're doing, they're not really that technically competent, I'm not going to invest". But since Netflix, Foursquare, and dozens of other companies were all offline at the same time, the investors were just like "they're using AWS, if Netflix can't keep online with hundreds of engineers, then I'm not going to expect that this little startup is going to do better". 7) Could I really do it better? Is anyone else doing better? AWS is the leader, so we all see the warts ... but at least they keep their certs updated (ahem microsoft)

Some more items:

o Liquidity of engineering talent - thousands of engineers know how to immediately work with the AWS infrastructure

o Design Patterns - They are constantly being updated, discussed, improved upon. People share how to work around AWS issues.

o Commitment to reducing prices - AWS is constantly dropping the prices for everything.

o Commitment to increasing the number of services - AWS is constantly releasing new and useful services on their infrastructure.

o Likelihood of being in business in 3 years. AWS isn't going anywhere. You can rely on them being around.

o Really good post mortems. Each time they fail, they do pretty transparently, and deliver a really painful post mortem that encourages me to believe they are committed to constantly improving.

o Multiple Availability and Region zones. I can, with very little effort, make my services available in redundant regions. I don't think there has ever been a multi-region failure with AWS, and, at most, it's only happened a couple times in their history. Multi-Availability zone failures have happened (and Amazon has apologized - that shouldn't happen) - but, it's < half a dozen times.

o Market Leader Advantage - The market leader tends to win. Microsoft for Desktops, Oracle for Databases, Linux for simple Web/Database/PHP stacks. This isn't always the case (Juniper can beat Cisco in a lot of places) but you know Amazon will be a Top-3 contender for at least the next 5 years.

I realize a lot of these are "Business" reasons, which is incredibly frustrating for people who are comfortable with hosting bare metal, and realize that, given just half the money, they could go into a CoLo, rent a cabinet, buy some servers, routers, firewalls, storage - install it, and return twice the performance that they would get on AWS.

But quite often, the business argument is a more important driver for CxO people making decisions (and assigning budget) than technical reasons.

And, with all that said - I've been in about a dozen Tier-1 data centers in the last 15 years, and, with one exception, every one of them has had multiple sustained outages. (The exception has been Switch SuperNap in Las Vegas - 3 years and no major outage.) - So, failures are part of the business. I'm not expecting Amazon to eliminate them, so it doesn't disappoint me when they happen.

Serious question: You have hundreds of hosts, but you don't think the savings of colocating your own hardware won't pay for an engineer to take care of the machines?

It would probably be possible to do it cheaper, but there is tremendous cost advantage at scale in this business. Then there's the question of focus. Do I want to have to hire the ops folks? their managers? create new payscales and review standards? Am I going to be good at attracting or judging talent in this field that is only tangentially related to software that I do know? Will the best people in that field want to work for me?

Serious question -- how are alternatives so much better?

You've always got to "waste your very precious time" on something -- putting together servers, dealing with hard drive crashes, etc. No matter what, you have to plan for things breaking, and a lot of the time, they'll break in ways you couldn't plan for.

It's not like there are other cloud platforms out there with zero failures and zero deficiencies. But how, for example, is SoftLayer better than AWS?

> But how, for example, is SoftLayer better than AWS?

Where do I begin... The latency to our dedicated DB servers is always 1-2 milliseconds. Never 5. Never 50. I do not understand how people tolerate 50ms response time from memcached servers on AWS.

We pay much less "per request". Way less. We never have to worry about servers disappearing. We don't need proprietary APIs for server images because the Chef script makes me a DB or a DNS box out of any Linux box.

We do not waste any time "architecting for failure": our databases run on RAID arrays with hot standbys, no need for chaos monkeys. Everything is very, very simple. We never used SoftLayer documentation because nothing of what we use needs to be documented.

And most importantly, we had less than an hour of downtime in 3 years. And none of that was Softlayer's fault.

It becomes much harder to reason about performance, uptime and responsiveness of your system when the underlying infrastructure does not provide guarantees on basic building blocks like I/O latency. Everything on AWS is always in flux and you have to over-allocate resources and/or over-engineer around those faulty building blocks. It may make you feel like a hero to survive in this jungle, but doesn't it look like a "forced premature optimization"?

99% of every graduating YC class could be running on a single pair of dedicated low-end servers and saving themselves a TON of time.

> We do not waste any time "architecting for failure": our databases run on RAID arrays with hot standbys, no need for chaos monkeys.

Ha - your in for some fun the first time your hardware fails.

We're not. Hardware failures were "invented" before Amazon marketing thought of them. There are reasonable engineering overhead to deal with them, and we do. You are welcome to go and unplug any of our severs with no measurable downtime or data loss.

My point is that we didn't have to spend nearly as much money and time as AWS forces you to do.

Another point I want to make: not all hardware is the same. If you're running in Amazon/Google style data center optimized around energy efficiency and real estate density and you're spending serious R&D money on tolerating server failures.

But guess what: you are not Google and you are not Amazon. You need to be in a datacenter which is optimized for individual server longevity. Cooler temperatures. Better hardware. Have a hot stand-by. That's cheap. Such way to "compute" may be expensive for Google scale to run, but it's much cheaper and easier for nearly everyone else.

Given your response - I don't understand why you would then make the statement about Chaos Monkey. It also appears that your earlier statement appears to be false: "We do not waste any time "architecting for failure"". If you can pull the plug on a server and not lose service, you certainly have architected for failure.

Chaos Monkey does effectively exactly what you claim to be able to survive - it unplugs servers. The fact that it runs on AWS is inmaterial, it exists to make sure you really can survive a server disappearing. The underlying point is that many teams build their software to survive server failures and never test the fail-over till it occurs for real, that often goes very badly. Chaos Monkey tests fail-over on a regular basis so you find out about it before it costs you hours of downtime.

I always point to this article: http://www.jamiebegin.com/why-an-ec2-instance-isnt-a-server/

> People, AWS original sales pitch was to take care of scaling for you, not the other way around.

As far as I know, it was never about that, but about giving you the tools to spin-up whatever you need at will. The cool thing about AWS has always been the APIs.

I would rather take a service that has downtime but give me the tools to failover/recover than a service that promises something they cannot promise (100% uptime) and when it inevitably fails, I'm at the mercy of their support.

Went from an AWS emergency every month or so to nothing but smiles with Linode. AWS is the cloud ghetto. Yes, Linode isn't a platform like AWS, but I was just using EC2 and all the platform tricks like the RBS and whatnot really are a lazy man's "I'm too busy to install mysql on a box" excuse.

People who need instant scaling have limited alternatives, but not sure why everyone is sticking with AWS.

Point me to someone else who does latency based routing (roughly equivalent to GeoDNS) with data centres around the world and I'm there.

>The world is full of wonderful alternatives

Because those alternatives as just as bad.

>We've been running a very high traffic system there with individual box uptimes of 700 days before we had to reboot them during scheduled OS upgrade/maintenance.

So, just like EC2?

Definitely not like Amazon US-East-1.

Which providers offer something equivalent to EBS, are worldwide, and have better reliability?

Worldwide is not everyone's requirement. Is Amazon reliable across the world? Sure. So is any other provider if they have more than one datacenter, or you use more than one provider and each provider is in different locations.

When the question is "why do people use X?", and people give totally reasonable answers, it doesn't make much sense to respond "but if we ignore your requirements then you could use someone else".

Reply on news.ycombinator.com