In summary, it is really hard to guarantee ACID across scalable, highly available, shared-nothing systems due to complex and high overhead commit protocols, and difficult tradeoffs in available replication schemes.
It's hard, but it's doable. I find it quite striking that no one ever seems to mention the two leading high performance parallel databases, HP Neoview and Teradata. Is it that people don't realize they exist?