First, we are getting better throughput from S3 than I we were using a SATA SSD. (and slower than a NVMe SSD). This is a bit of a secret.
Of course, single sequential throughput on S3 sucks. At the end of the day the data is stored on spining disk and we cannot do anything against the law physics.
... but we can concurrently read many disks using s3. Network is our only bottleneck. The theoretical upper bound on our instances is 2GB/s. On throughput intensive 1s query, we observe an average of 1GB/s.
Also you are not accounting for replication. S3 costs include battle tested, multi-DC replication.
Last but not least, S3 trivially decouples compute and storage. It means that we can host 100 different indices on S3, and use the same pool of search server to deal with the CPU-bound stuff.
This last bit is really what drives the price an extra 5x down for many use case.