Anybody care to comment on using EC2 with local (what Amazon calls ephemeral) storage and backup to S3? Seems to me the advantages are: it's cheaper and you avoid the performance and reliability problems with EBS. The disadvantages?
What do you tell the hordes of angry redditors when the last thirty minutes of carefully (or angrily) composed comments vanish?
Transferring 100gb+ of data on EBS (even with an 8x RAID) is a nightmare. Ephemeral drives, however, it's fairly fast.
Throw 100+ database connections at a few ephemeral drives (even in a RAID) and watch your web site slow to a crawl.
From http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs..., "On a good day, an EBS can give you 7,000 seeks per second and on a not so good day will give you only 200."
The ephemeral store will never give you more than a hundred seeks per second. If you're seek-bound, then EBS, every time.