Has anyone had good or bad experience running a high IO operation using EC2?
Those databases are all on XLarge instances, so there is minimal sharing, and we've also gone to great lengths to make sure all of our normal queries are in indexes, so the disk gets hit less.
We also have a read slave for every database to alleviate read loads.
One thing you might want to do is run 'iostat -xtc' on your current box and put that in a log file. Then go back and analyze it and see what your average and peak reads and writes are. Amazon's max for a single EBS appears to be about 1000 ops / second (at least, that is what we were doing when they told us we maxed out the performance of the disk).
Good luck!
Edit: I forgot to mention that on all the database disks, we use ext2 and noatime. Both decrease the total number of writes necessary, and have very little downside (the biggest being that you have to fsck on a crash).
I should mention that I'm running Windows with MSSQL. I imagine the performance should be the same but there must be some differences of course.
I like the option of rackspace allowing for a dedicated server next to the cloud but I should probably give EC2 a few test runs of my own too. I think if EC2 is going to work it wont be an overnight solution. I'll have to move over slowly, learning to tune the box as I use it more.
Actually, you might want to try and get in on Microsoft's Azure beta.
As far as Windows Azure being early in its life, you are correct. However, it has been in beta for about a year and it is running Windows Server 2008, which has been in the market for almost two years. You may also want to look at the Windows Azure forums at http://bit.ly/MSDNWinAzureForum to get an idea of other people's experiences with it.
You can also get more info about Azure and its beta availability at http://bit.ly/WinAzurePlatformDevCenter
(Jason - working for M80, representing Microsoft)
All of our MySQL instances are currently each running on 4x EBS volumes using LVM stripped across all 4 volumes.
Our primary transactional table is nearing a billion rows, and just this weekend we performed a migration that segmented this table into 40 partitions, containing data back to 2007.
We are currently handling roughly 3,000,000 inserts and 2,500,000 updates on this one table daily. IOWait on our DB holds steady at about 5% throughout our peak (5am - 4pm PST).
It is certainly not Fibre Channel backboned speeds, but the other benefits of using EBS CURRENTLY outweighs any IO performance hits we might have. The ability to backup production and refresh any of our other DB environments in less than 15 minutes, using EBS snapshots is worth it. Especially for the price we pay monthly for this infrastructure.
If you need any more details on our setup or tips in troubleshooting/setting up your own, feel free to ping me.
We explored all the options available and worked with both Percona and Amazon on the issues, but it was clear that EBS and EC2 just are not meant to handle such load. Our database is currently back running in our datacenter.
There have been a number of reports with similar results using EBS. It all depends on your particular IO profile, but since you said it was high, I encourage you to tread lightly.
If you want more information you are welcome to email me.
Our setup was surely not ideal, but even with significant tuning, it was not sufficient.
It is really good to hear successes like Reddit's, though. How long have you folks had your databases in EC2?
That's been a consistent theme I've seen with EC2 instances--it's hard to predict how fast something is going to be, and once it's running, you don't know if its performance is going to change. That's one reason to take benchmarks of EC2 (and possibly other cloud providers) with a grain of salt.
I've heard anecdotes of people starting up n EC2 instances, running benchmarks on each, then killing all but the fastest ones.
Anecdotally, Rackspace Cloud Servers I/O works much better, although the only serious head-to-head comparison I have seen is this one: http://pl.atyp.us/wordpress/?p=2240 (tl;dr: < 20MBps on writes with local storage or EBS on EC2 vs 100MBps+ on RCS)
According to Amazon the I/O depends on the instance type (http://aws.amazon.com/ec2/instance-types/) so the OP may want to take that into consideration when picking an instance type.
What I intended to say was "you can make Cassandra work on EC2." However, if the I/O throughput of EC2 is a fraction that of some other provider, you'll need more machines in EC2 to achieve the same throughput.
My personal site is hosted on the Rackspace Cloud and I've been very pleased with it, but my workload is different so I can't give a comparison.
I also know that we've benefited directly from jbellis' support on IRC and the mailing list, so this should be considered an endorsement of Cassandra, if anything.
That said, here is a good writeup on RAID use with EBS by someone at Heroku: http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs...
These cloud services seem great for low IO apps but once you have to hit the database hard it seems like anyone's guess if your db will perform.
My understanding is that Github virtualizes things themselves, they're not running Rackspace cloud servers.
Also, if you have a part of a DB that just requires unholy amounts IO, there is of course the new 68gig instances, move that part and run it purely on memory with a combination of frequent dumps to disk.
http://docs.amazonwebservices.com/AWSEC2/2007-08-29/Develope...
My opinion of what constitutes "middle of nowhere" may differ from the average San Franciscan's, but I've had the opposite estimation of costs that jedberg did. That is, EC2 was 20% more expensive than operating ones own hardware in a South Bay datacenter for a year.
This would have been about a year ago, but, IIRC, including operating costs, one would break even at around 10 months of unreserved EC2. Pricing has certainly changed in the meantime, so 30% in the other direction is well within the realm of possibility.
Back then, it cost about $3 per year per watt. Now it's pushing $4. The trick is, in the case of ones own hardware, one usually pays for the available capacity, not just what one uses. On a 500W server, that's $2k annually.
That's the flexibility that AWS offers: one doesn't pay for the operating capacity, only the use. It would seem, however, that one pays for the hardware up front either way.
The great increase in flexibility of operating ones own hardware is not suffering from only ordering off an abbreviated menu. Germane to the OP's question, I have never found any virtual or dedicated server offerings come close (say, a factor of 10) to the I/O performance of something custom configured/assembled (even from inexpensive, commodity blocks). Whether this constitutes a problem is subject to interpretation
In general, I remain skeptical that there exists in The Cloud cheap-enough commodity servers suited to even a majority of applications. Remember, even Google had custom hardware very early on.
If the capacity you need is fixed and known it will be cheaper to get a dedicated server. Here's a a counter example. I once worked for an online retailer and we got an order of magnitude more traffic at christmas. With a dedicated server we had to pay for the capacity needs of christmas all year. On the cloud we could add capacity for xmas and then turn off hte extra capacity the rest of the year. It was a lot cheaper.
Also, I was comparing against 1 year reserved instances, which is what I use now.
In your "IAMA" thread on reddit, someone estimated that Reddit must be pulling on the order of $1M per month, making the costs neglibile. Digg must be earning even more, because of its greater number of users and their higher susceptibility to ads.
I use the database as a raw data source and have a cron that generates read only data sets for the web servers to use. This works for my application, but probably not most.
You can Raid EBS devices btw.