Latency I can see. I find it hard to believe S3 has a lower availability than a sata drive. Given that S3 is replicated.
And we haven't even started to talk about eventual consistency.
I am the author of goofys (which exposes s3 like a filesystem) so obviously I believe this can work, but you do need to understand the tradeoffs (or you use goofys and hope that the tradeoffs I've made are correct :-P)