Amazon EC2 offers 68GB RAM and it's hardly an obscure offering. But they don't offer SSDs.
SSD might be a solution, but we don't have any proof either way, and we also don't understand why is this problem happening.
SSD might be a solution, but we don't have any proof either way, and we also don't understand why is this problem happening.
SSDs really help this process happen faster because they have way better random write performance than hard disks.
=============================
What to do?
I wish I could say you should use Percona Server with XtraDB.
If we were using SSD as storage, then I would recommend it. Vanilla MySQL performs equally bad on SSD and HDD, while for SSD in Percona Server we have “innodb_adaptive_flushing_method = keep_average”.
Unfortunately on spinning disks (as in this case), Percona Server may not show significant improvement. I am going to followup with results on Percona Server.
=============================
SSD is the solution because unlike rotational disks it allows for lots of random accesses. While they are too expensive to use as "archiving" storage, SSD excels as a layer between traditional disks and RAM. Put your high I/O workload on SSD and keep your historic data on spinning disks.
If you want to see some SSD benchmarks, here is your resource: http://www.ssdperformanceblog.com/
As far as cloud storage, yes, Amazon EC2 offers 68GB RAM, however if you need hardware of that size it is not hard to justify setting up your own servers. It would be cheaper too and you'll be able to fully optimize your storage.
My macbook pro I upgraded with a gaming grade SSD gets 100-200 mb/s random write.
Just upgrading to SSDs is more viable than hacking up a database with a huge community behind it that you didn't originally create yourself.