NVMe read latency is in the 10-100µs range for 128kB blocks. S3 is about 100ms. That's 3-4 OOMs.
The threshold where the total read duration starts to dominate latency would be somewhere in the dozens to hundreds of megabytes, not kilobytes.
This keeps the new caching layer simple and take advantage of the existing caching. If they went any bigger they'd likely need to rearchitect parts of the keymap or underlying storage layer to accommodate, or else face unpredictable TCO.
Aren't you comparing local in-process latency to network latency? That's multiple OOM right there.