This experiment feels similar to people who buy old cars and remove everything from the inside except the engine, which they tune up so that the car runs faster :).
More importantly it is about the idea of using tools like Flamegraphs (or other profiling tools) to identify and eliminate your bottlenecks. It is also just fun to experiment and share the results (and the CloudFormation template). Plus it establishes a high water mark for what is possible, which also makes it useful for future experiments. At some point I would like to do a modified version of this that includes DB queries.
YouTube or Google Search suggestion is good, and I think it could be replicable with that amount of data. What is insane is the speed. I can't think how they do it. I am doing something similar for the company I work on and it takes seconds (and the amount of data isn't that much), so I can't wrap my head around it.
The point is that doing only speed is not _that_ complicated, and doing some algorithms alone is not _that_ complicated. What is really hard is to do both.
With sufficient caching with and a lot of parallelism makes this possible. That costs money though. Caching means storing data twice. Parallelism means more servers (since you'll probably be aiming to saturate the network bandwidth for each host).
Pre-aggregating data is another part of the strategy, as that avoids using CPU cycles in the fast-path, but it means storing even more copies of the data!
My personal anecdotal experience with this is with SQL on object storage. Query engines that use object storage can still perform well with the above techniques, even though querying large amounts of data from object is slow. You can bypass the slowness of object storage if you pre-cache the data somewhere else that's closer/faster for recent data. You can have materialized views/tables for rollups of data over longer periods of time, which reduces the data needed to be fetched and cached. It also requires less CPU due to working with a smaller amount of pre-calculated data.
Apply this to every layer, every system, etc, and you can get good performance even with tons of data. It's why doing machine-learning in real- is way harder than pre-computing models. Streaming platforms make this all much easier as you can constantly be pre-computing as much as you can, and pre-filling caches, etc.
Of course, having engineers work on 1% performance improvements in the OS kernel, or memory allocators, etc will add up and help a lot too.
It's hard to measure, so nobody does.
Throughput is easy to measure, so everybody does.
Latency is hard to buy, so few people try.
Throughput is easy to buy, so everybody does.
Latency is what matters to every user.
Throughput matters only to a few people.
Turn on SR-IOV. Disable ACPI C-states. Stop tunnelling internal traffic through virtual firewalls. Use binary protocols instead of JSON over HTTPS.
I've seen just those alone improve end-user experience tenfold.
Related, I worked on an embedded linux device that bit-banged an unusual communication protocol. I wanted to offload this bit banging to some dedicated microcontrollers built into the SOC, to help free up the main CPU for some other use.
I spent some time writing a nice little device driver that used DMA to send the transaction to the bit banger. The main CPU usage went from 60% down to 2%! Success! Unfortunately, the transactions/second was now a fraction of what it was before.
After some digging, it turned out the improved CPU usage was low enough that the CPU scaling governor was staying at the lowest frequency. Disabling the governor and forcing the CPU to max frequency resulted in a pretty huge transactions/second improvements. Obvious in retrospect, of course.