> all (not really all) I/O goes to their cluster FS (Colossus, sort of).

Well - sorta. Cluster FS just means disk I/O is now network I/O (EBS is a bit like this). Now you push the hairy problem to a different area: your network topology starts to matter a lot. You might end up having to think about not scheduling too many I/O heavy jobs behind the same switch or something. AND disk I/O is now competing with network I/O. AND it might now be hard to predict the destination of where all that I/O is going to be when scheduling the job, whereas previously it used to just be local.

So the problem doesn't go away. But maybe you say it's just easier to wire everything up with gigE / infiniband so that you're massively under-capacity, and call it a day, and that might be a fair tradeoff.