Because there's an extra virtual layer between you and the hard-disk, fsync() does not do what it says it does, which means that a database cannot actually guarantee that a transaction was successfully completed.

In case your traffic is huge, this can cause big problems that are very hard to fix. And it did cause problems for Reddit, as EBS is notoriously awful in this regard (and others, like big latencies on access).

Basically when working at big scales, nothing beats having complete control over your infrastructure. It may be tough and time consuming, but at least you can identify the problem and fix it.

It is my opinion that something like Google's search cannot be built on top of Amazon's AWS.