"S3 costs include battle tested, multi-DC replication."

Sometimes we pay a bit too much for this multi-replication, battle tested stuff. It's not like the probability of loosing data is THAT huge. For the 4x extra cost you could easily take a backup every 24h.

"It means that we can host 100 different indices on S3, and use the same pool of search server to deal with the CPU-bound stuff"

You can do that with NFS.

It's amazing how much we are willing to pay for a bunch of computers in the cloud. Leasing a new car costs around $350/month. You could have three new cars at your disposal for the same price as this search implementation.