How is it that the S3 API is remotely fast enough to make this work?
As search engine that operates at any kind of scale needs to skip through very large files to evaluate a query. You need very low-latency, high-bandwidth access to disk. A search engine instance that accesses files on a local SSD is an order of magnitude faster than one that puts files on EBS.
They make some mention of local caching, but the devil is in the details here. Does all data get copied to local cache? What is the performance here?