Comment by karterk | Hacker News Reader

Comment by karterk | Hacker News Reader

Parent Full thread

This article pretty much reflects my experience with EC2. EC2 makes it very easy way to get started with things when you're figuring stuff out and don't have a great deal of experience in managing servers and machines. You don't have to deal with RAID, power outages, etc. and can literally treat hardware like software.

But once you get up and running, you will realize that this comes with a great cost. EC2 absolutely sucks in terms of raw performance. I also notice variation in performance of machines on different times of the day, and between machines.

Amazon has done a great job at selling "the cloud", and I can see many CXOs buying into that. But the fact is, renting dedicated machines from a good provider isn't exactly that hard. They take care of a lot of things for you.

EC2 is good if you have wildly varying amounts of traffic such that you need to really have that kind of elasticity in your infrastructure. However, for most businesses, especially web apps, that's not the case.

operations is hard, let's go shopping!

I was shocked when I learned that many EC2 users with highly variable traffic don't spin down during off-peak because sometimes Amazon has problems that prevent them from spinning their instances back up.

I love the idea of their cloud services, but the reality is a decidedly less attractive beast.

Hmm I would like to hear more about these problems that prevent people from spinning up instances. Is it a frequently occurring problem or only happens rarely (e.g. when APIs are down). Also are they managing the instances themselves or using EC2's AutoScaling cluster? I run a dynamically scaling cluster on EC2 and have not run into the problems like you mentioned, so I would like to hear more about them if possible. Maybe they are spinning up and down too rapidly and exceeds the API rate limit? You'd have to have a bugged/bad provisioning system to accomplish that though...

In the past, when an AZ goes down, everyone tries to transfer to another AZ and makes lots of API calls.

Consequently AWS has to rate-limit API requests to avoid taking another AZ down.

Maybe this has changed though, but we'll probably have to wait for another disaster to know!

---

But of course, an AZ shouldn't be your infrastructure's single point of failure!

I'm aware of the API rate limit when an AZ goes down, but the original comment was about problems that people run into for highly-variable traffic sites that scale up and down on a regular basis (e.g. spin up 20 instances during the day with high traffic, at night shutdown 15 of them to save cost, on a daily basis). This of course is not related to any AZ/API downtime, and I am not aware of any problems that could interfere with the normal usage of APIs during normal service operations, which is why I wanted to hear more about the details of those problems.

It's due to rarely occurring problems such as APIs down/unresponsive, insufficient/incorrect instances available, and a third major class of problem that's frustrating my memory at the moment. In addition to not scaling up and down (automatically or manually), many of these apps are architected to require as little from AWS as is possible to reduce their exposure. For example, they'll refrain from using ELB because ELB depends on EBS.

These decisions were made by companies that started off believing fully in the promise of elasticity, and gradually shifted to less elastic architectures as they experienced issues. That said, it's worth noting that these are firms with very high costs of downtime, so the magnitude of failures was very high.

Reply on news.ycombinator.com