How to Read 100s of Millions of Records per Second from a Single Disk

Hard-disks aren't slouches when it comes to sequential throughput, but building around disks has some serious non-future-proof consequences.

Building things for spinning disks is a cul-de-sac, which has no exit over onto the faster characteristics of SSDs.

A significant part of Hadoop is built around the idea that "random = bad, sequential = good" and this is still arguably true when it comes to writes.

However, building around sequential reads gives you the wrong idea when the hardware changes underneath you.

I see a tiny sign of that in this post, where it talks about the smarter data partitioning.

For example, when dealing something like a join, a spilling sequential write join like Grace hash-join works much better than a memory mapped hashtable for spinning disks - however a grace join will be crushed by a better random IOPS driven SSD.

Also, if the author is reading this - Zlib isn't a slouch either (just like a spinning disk), the trick is to use the right combination of Zlib parameters. Compression will be slower with Zlib, but the read throughput of Zlib can exceed the performance of memcpy of the same data depending on your data conditions (look for inflate_fast and inflate_slow).

And if you're getting into the business of improving sequential reads, the next big trick for systems is vertical row partitioning of data (think of TPC-H, except now you stored L_COMMENT in a separate file).

This seems like a very clever way to accomplish this feat. I think I'm happy that to date I haven't the sort of problems that might require reading 100M records per second. However, I'd be all ears if someone wanted to contribute to my education by telling me what sorts of interesting things that this might enable.