Hacker News Reader

Comment by powdahound | Hacker News Reader

Make sure you explore the limits of EBS before assuming it's a perfect solution. We've found it to have incredibly slow throughput at times.

Some reference links:

  - http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs
  - http://victortrac.com/EC2_Ephemeral_Disks_vs_EBS_Volumes
  - http://stu.mp/2009/12/disk-io-and-throughput-benchmarks-on-amazons-ec2.html

View on HN

I've also seen highly variable io performance (ie, slow/stuck) on EBS volumes. Sometimes, all reads/writes block for several second or minutes, with iowait going to 100% on an m1.small.

example: http://developer.amazonwebservices.com/connect/message.jspa?...

However, I've decided that the other benefits of EBS volumes are just to important to give up on (snapshotting, lazy-load, re-attachment). Instead, I plan to monitor for such situations and blow away the node when I detect it.

But really, I don't understand why Amazon doesn't fix this. It's happened to me 3-4 times in a relatively small installation. Surely their monitoring can detect this?

Is it because of the network throughput on the small instance type? I haven't noticed anything nearly that severe with our 20-instance setup of larger instance sizes, but then again we've only run production on EC2 for a few months now...

But really, I don't understand why Amazon doesn't fix this.

When you select your instance-type then you get to choose between "Low", "Moderate", "High" and "Very High" I/O performance.

Want to take a guess at which instance types suffer first when there is high I/O demand or contention?

I can handle suffering, but completely freezing?

That is likely not intended so I guess it's probably inherent to how Xen and/or their I/O layer do the resource allocation.

I, too, have seen these freezes occasionally on small instances. If you think that is harsh then try one of the new micro's. ;-)

Anyways, I haven't had such a freeze last longer than a minute yet, and the instance would always recover. If yours does not recover then that would clearly be a bug.

As a heavy user of EBS, I heartily concur. The ability to move EBS volumes around is handy but you pay it back in inconsistent, poor-to-mediocre performance. Good luck.

We had these issues as well.

I'd certainly try AWS, but I'd wait a couple of months before shelling out the upfront cost for the Reserved Instances.

EBS definitely has performance limitations, however creating striped RAID arrays of EBS volumes significantly helps with performance.

This blog post walks through a bunch of things you can do to make the best out of the flexibility you get with EBS, and includes scheduler tuning, RAID block size planning, etc.

http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs...

My team runs a large AWS/EBS backed implementation and, while we've seen some of the EBS performance issues, we've certainly engineered past them to get us where we need to be.

Wouldn't creating striped RAID arrays of EBS volumes add to the cost and remove the ability to do snapshots? How do you get a consistent snapshot of multiple striped EBS volumes?

It just makes it more complicated. You have lock and snap all of the volumes at once. The ec2-consistent-snapshot tool makes this easier to manage.

EBS costs are pretty low. And my above post was directed to someone who is deploying a database on top of EBS. Sorry if I didn't make that clear.

In our case, we replicate to set of non-RAID EBS instances where we pull snapshots.

Also keep in mind that creating a RAID0 of EBS volumes dramatically reduces your reliability too. If any one of the underlying EBS volumes misbehaves the entire RAID set has problems.

Snapshots are still doable, they just require coordination of all the underlying devices to make sure they are all in a consistent state. The way you do this can vary but it's not particularly hard.

Reply on news.ycombinator.com