How many people have dedicated hadoop clusters available to them in the first place? The point of the pattern, framework and product is to scale linearly and on demand, not to do the fastest computation per-machine possible.

As to IO - HDFS is going to divvy up the workload from S3 when the cluster bootstraps. After that the data will be local to each compute node, so they won't be getting kicked in the face. There is no reason to suppose that S3 replaces HDFS - its just used to load your data in.

Also, one would hope that their Hadoop setup IS rack aware - why shouldn't it be?