The title should be “10x engineer creates Google in their spare time”
But seriously what an amazing write up, plus animations, analysis etc etc. Bravo.
It was also ironic to see AWS failing quite a few use cases here. Stuff to think about.
But seriously what an amazing write up, plus animations, analysis etc etc. Bravo.
It was also ironic to see AWS failing quite a few use cases here. Stuff to think about.
> SQS had very low concurrent rate limits that could not keep up with the throughput of thousands of workers across the pipeline.
I could not find this perhaps the author meant Lambda limits?
> services like S3 have quite low rate limits — there are hard limits, but also dynamic per-account/bucket quotas
You have virtually unlimited throughput with prefix partitions
I definitely wanted these to "just work" out of the box (and maybe I could've worked more with AWS/OCI given more time), as I wanted to focus on the actual search.