Greplin (YC W10) AWS Benchmarks and Best Practices

Thanks for putting this together. A few questions come to mind:

1) Did you try bigger instance types, i.e. large? I've heard at that level and above you get better networking performance, which theoretically might improve EBS performance?

2) Related to #1, I wonder if you've tried doing RAID0 on the ephemeral drives? You get two with the large instance, and four with xlarge.

3) Did you ever measure non-IO network EBS performance during the tests? I've always wondered whether using EBS heavily would slow down other network traffic to the device given there is only one interface.

4) How often have you yourself experienced EBS volume failure in your RAID volumes?

5) When that happens, what happens to your volumes and instance? That is, what do you use to monitor when the RAID volume degrades? Does it usually take down the instance immediately or only after some time? If it just becomes really slow does that throw alarms or does the application just become really slow?

6) Finally, what is your current procedure for dealing with a volume failure?

Well that ended up being a lot of questions. I'd really appreciate any answers/insights your or anyone else could shed on these questions. I've been reading all these posts but it feels a bit like reading tea leaves.

Great test. I wonder if concurrent db access(submit db reads to multiple db servers and pick the first one) would enable workaround these weird variations in EBS response times.

Adrian C from netflix had a nice tech article on EBS performance. "Understanding and using Amazon EBS - Elastic Block Store" http://perfcap.blogspot.com/2011/03/understanding-and-using-...

Problem with RAID is not just drives failing, but also the UBE (Unrecoverable Bit Error). Lets say you have a RAID5 configuration. In addition to the probability of disk failure, you need to account for the probability of some failure plus the controller's inability to repair the problem because a sector on the good disks experienced an UBE. The probability of UBE on enterprise level disks is rather low (10^-16 I think) but that quickly shoots up if you start considering 1TB and 2TB drives.

Problem with EBS benchmarks is that they largely depend on who else is sharing the spindles with you at the time the benchmark was run. Given the large variance in performance that is being reported, the sample size for some reliable statistics would need to be quite large

"RAID helps smooth out flaky performance"

Really? My experience with EBS has been more to the opposite... especially with write performance.

If you got to write data, you got to write data. Flakey performance becomes more likely as more volumes are added.

Previously I think AWS engineers recommended we use RAID0 with EBS... ugh. Don't care cloud or not, RAID0 on production DB servers sounds downright suicidal.