Hacker News Reader

Comment by bob1029 | Hacker News Reader

I've achieved over 2 million inserts per second (to SSD) using a simple splay tree, append-only log, and a ring buffer abstraction (for serializing access).

Writing in batches of sizes dynamically determined by the backpressure mechanism seems to bring incredible performance benefits. Both SSDs and CPUs benefit hugely from any technique that can batch transactions at any level.

To the user of your database, what's the difference between 1-10ms best case, or 5-20ms average? To your CPU and flash, that 10ms extra batch width can mean the difference between 5000 and 500k inserts per second.

View on HN

This suggests you don't commit transactions individually, but buffer them for some time and then write them to disk at once. There are plenty scenarios where you don't need all ACID properties, but in many cases what makes databases slow is adhering stringently to those ACID constraints.

Correct. I can achieve multiple transactions per I/O. I do not believe this actually breaks any ACID properties. Reads and writes are serialized to a single thread for all processing.

>I do not believe this actually breaks any ACID properties.

You must send the write ACK after your transaction has been written, which could increase latency in the worst case but you will still have much higher throughput.

ACID is what makes it possible to write parallel applications and stay sane. I'm always in awe when I realize how crucial it is.

it's often not particularly crucial for timeseries data, though, and the performance tradeoffs of syncing every insert to disk are significant.

yes I was seeing from my CRUD applications point of view :-)

Is there any stable application for this? I do have an application with varying very high and very low incoming data packets I need to save. They can be really small json documents or have larger files attached (up to 10mb). Kafka would be a perfect fit, but the file size is too big, and saving every attachment first to a cloud storage bucket will cost a lot because of many STORE commands (i would like to merge the files before storing to reduce the cost).

I am using it as the backend for a custom database engine. Key-value is the foundation. When you can transact millions of them per second, it gets easy to try crazy new things on top.

Not really used to doing low level programming but how would you assess back pressure for I/O ops on SSDs or disks?

Are you only evaluating running OPs rates compared to a theoretical maximum?

> how would you assess back pressure for I/O ops on SSDs or disks

I am not attempting to do anything like this. The back pressure is an inherent property of the ring buffer abstraction I have selected for serializing transactions.

Batch size is effectively determined by how long the previous batch takes to complete. So, if one batch takes a little while, but is not at RingBufferMaximum, the next batch will be a bit larger (assuming constant system load), thus extracting a little bit more uplift from the benefits of running everything on hot cache lines. The maximum size of the ring buffer prevents things from getting out of control. You simply test for the best batch size and set that as the max for the buffer.

Best case scenario, your system is fully loaded and slamming through 4k+ batch sizes all day long. Worst case, you are doing a transaction per IO. I.e., what everyone else on earth is doing right now.

Ah right, I see. Thanks!

Any resources you would recommend about this kind of low level I/O programming and tuning?

Reply on news.ycombinator.com