Assuming you use InnoDB, is your innodb_buffer_pool_size big enough to hold your data (or a significant amount of it)? It's by far the most important setting to get speedy reads, which form 95% of the queries in our case. When you did the testing, did you warm up the RDS instance before, for example by running a number of common queries? It naturally takes a while after booting until MySQL can make efficient use of caches and buffers.

I'm not sure if instance-local storage will give you better performance compared to EBS. From the EBS page: "The latency and throughput of Amazon EBS volumes is designed to be significantly better than the Amazon EC2 instance stores in nearly all cases."

There is a problem with the recovery approach you described: as far as I know you can't use an EBS snapshot to populate local storage on a new instance, so you would be forced to use EBS for the new master. Promoting the slave to be the new master would be the better approach, with less downtime and data loss, provided that MySQL replication didn't break unnoticed.

- Tobi