With Qubole you can offload data engineering to their platform. Cluster management is super simple. Hand rolled solutions in my experience are a pain and elastic cloud features take up time to build. Qubole's offering provides out of the box experience for most big data engines out there. Presto/ Spark/ Hive/ Pig - what have you - all work with your data living in S3 (or any other object storage). I believe they have offerings in other clouds too.
Some amount of S3 listing optimisation is done by Qubole's engineering team for:
https://www.qubole.com/blog/product/optimizing-s3-bulk-listi...
They also have features that allow you to auto-provision for additional capacity in your compute clusters as your query processing times increase.