AWS: the good, the bad and the ugly

Can you elaborate on this a bit? Why are you scared of SNS? Data loss / latency etc?

FYI http://www.1000tb.com/ is the website for a landscaping company, not about bandwidth...

We switched to Elasticache recently as a test, and found that it works fine but don't see any compelling reason not to just manage our own memcached servers.

Edit: Also, what scares you about SNS? We use it for our oncall alerting and have seen no reason to be concerned about reliability - but I'd love to know if we should be!

Disclaimer: I work at SendHub, but I've done it both ways and I definitely prefer ours ;)

Intel E3-12303.2 GHz8GB2 x 1TB100 TB$201.15 1Gbit Dedicated Port

So, for $200/month, they'll give me 100 Terabytes/bandwidth on this server.

250 megabit/second at 30 days in terabytes = 81 Terabytes. A decently peered/connected Pipe costs, at this volume, around $5-$7/megabit @95th, or $1250.month.

So either:

  A) Their connectivity isn't hot.
  B) If you actually use that 100 Terabytes per server, 
     they start to curtail/rate limit/traffic shape you.
  C) Some other option I haven't considered?  Maybe they 
     just rely on the vast majority of their customers not 
     using that bandwidth, and subsidizing (significantly) 
     those that do?

In my experience, with large data sets (I'd say > 100 GB), you want to run your own database instances, if only so you have control over the small fiddly bits you need control over.

RDS: 1) Uses drbd for master failover (this introduces network overhead on disk writes), 2) Only gives you access to super commands through stored procedures, and 3) doesn't let you tune InnoDB to your dataset or performance requirements.

The point in the post that the AWS management console runs on EBS is fishy...

As parent says: EC2 does have its place, but for deployments in the 10-20 hosts range (or an app with "special needs") you tend to pay through the nose. Keep in mind that the amount that you pay for a fraction of a server on EC2 will rent you the entire box elsewhere.

With network filesystems (eg. NFS) you can choose to return an I/O error to the application when you hit a timeout or a network error (the -o intr mount option). This is rarely used since applications aren't used to dealing with them. So can't really blame the OS here either.

Also, do you backup your databases anywhere other than on other ephemeral instances?

Our ops guy is much better than me, but his time is better spent working on higher-stack stuff like deployment automation, monitoring and efficiency tuning than on re-inventing a virtualization stack to save a few thousand dollars every month.

If we were bigger, it would be more worth the time and money spent. But without doing the math, we would have to be quite a lot bigger, I think.

And you can find essays of people who run their own rack who swear the same thing about moving to AWS.

I worked at an all-AWS bay area startup that ran a $100k/mo Amazon bill, and now I work at a bay area startup that runs all its own hardware.

It's about tradeoffs. It's not just about cost. (Though there are many cases where Amazon is cheaper and it has little to do with your monthly spend, and certainly not an arbitrary threshold like $3k/mo.)

Writing this at the close of my employer's Open Enrollment period, I'd say this: Comparing AWS with running your own hardware is like trying to compare two health insurance plans. Each brings its own seemingly impossible tradeoffs. All you can do is bring your experience and judgment to identify core issues, make the best call you can, and suppress the urge to dream of the road not taken when you're knee deep in whatever hell you're experiencing that would be a non-issue if only you had went the other way.

Though if you loathe the AWS "fanboyism" so much then maybe it's the company you're keeping: Few currencies in a startup are as valuable as flexibility. The younger the startup, the truer that is.

In terms of AZs, were are distributed roughly evenly across all AZs in east-1.

However, our primary strategy for uptime is redundancy -- every db has at least one slave, and we are spread across multiple availability zones.

Stuff not in a database goes on S3.

Thus, most of the time, performance issues with EBS bite you when you have application data on an EBS volume. If you're constantly hitting the volume to serve data to customers, you'll be seeing the hiccups in EBS performance and passing them right on to customers. Clearly this is a suboptimal experience, and can lead to other failure modes; a slow/stuck bunch of ops can get an application to completely stall.

It's important to note here that EBS has no error reporting system or timeouts; common operating systems aren't very good at handling disk IO errors, so EBS will never produce them, even when its having issues. This lack of transparency can make handling/working around stuck EBS operations nigh impossible.

So, having experienced these pains, people generally say "Don't constantly use EBS if you care about your application performance/stability," and they aren't wrong.

That said, if you use EBS as a boot volume only, you are likely not hitting that volume enough to feel the pain of slow-IOs; once booted you won't touch it much. At the same time, using EBS as a boot volume has all kinds of conveniences: the ability to start and stop the instance without loosing data, snapshots and AMI creation, persistent machine-specific configuration, etc.

I hope that's useful. EBS is not a perfect service, but it definitely has its uses. EBS as a boot volume has made using AWS quite a bit friendlier.

Note: I worked on the EBS team at AWS a few years ago. I currently use EBS in my startup's infrastructure. If you have further questions, feel free to reach out to me.

Many times what I've seen is simply that position is simply 'filled' by developers whom have ideas about designing and securing scalable systems, but when it comes right down to it... they get hacked within some period of time because it turns out, they don't know these things well enough to take the place of someone whom dedicates their time to it.

AWS's security groups pretty well cover you on the firewalls front (although some folks like to still run software ones on their instances to be doubly sure), but you're otherwise correct.

For a single instance, you're not going to see significant differences. It's running large clusters of instances you're going to see things being better than the average VPS provider, with stuff like Virtual Private Cloud etc.