Comment by bob1029 | Hacker News Reader

Comment by bob1029 | Hacker News Reader

My take on NVMe has evolved a lot after getting a chance to really play around with the capabilities of consumer-level devices.

The biggest thing I have realized (as have others), is that traditional IO wait strategies don't make sense for NVMe. Even "newer" strategies like async/await do not give you want you truly want for one of these devices (still too slow). The best performance I have been able to extract from these is when I am doing really stupid busy-wait strategies.

Also, single-writer principle and serializing+batching writes before you send them to disk is critical. With any storage medium where a block write costs you device lifetime, you want to put as much effective data per block as possible, and you also want to avoid editing existing blocks. Append-only log structures are what NVMe/flash devices live for.

With all of this in mind, the motivations for building gigantic database clusters should start going away. One NVMe device per application instance is starting to sound a lot more compelling to me. Build clusters with app logic (not database logic).

In testing of these ideas I have been able to push 2 million small writes (~64k) per second to a single Samsung 960 pro for a single business entity. I don't know of any SQL clusters that can achieve these same figures.

In NVME you can get around 800.000 IOPS from a single device, but the latency gives you around 20.000 IOPS sequentially. You need to talk with deep queues or with multiple concurrent threads to the device in order to eat the entire IOPS buffet.

Traditional OLTP workloads do not tend to have the concurrency to actually saturate the NVME. You would need to be 40-way parallel, but most OLTP workloads give you 4-way.

Multiple instances per device are almost a must.

> You need to talk with deep queues or with multiple concurrent threads to the device in order to eat the entire IOPS buffet.

Completely agree. There is another angle you can play if you are willing to get your hands dirty at the lowest levels.

If you build a custom database engine that fundamentally stores everything as key-value, and then builds relational abstractions on top, you can leverage a lot more benefit on a per-transaction basis. For instance, if you are storing a KVP per column in a table and the table has 10 columns, you may wind up generating 10-20 KVP items per logical row insert/update/delete. And if you are careful, you can make sure this extra data structure expressiveness does not cause write amplification (single writer serializes and batches all transactions).

You may want to play with a TiDB setup from Pingcap.

  > If you build a custom database 
  > engine that fundamentally stores
  > everything as key-value, and then
  > builds relational abstractions on
  > top

Sounds like this could be FoundationDB, among other contenders like TiDB.

https://foundationdb.org

More like myRocks. FoundationDB doesn't use LSM and definitely wants to do lots of overwriting in place. TiDB uses Rocksdb and would be closer.

With a lot of NVMe devices, up to medium priced server gear, the bottleneck in OLTP workloads isn't normal write latency, but slow write cache flushes. On devices with write caches one either needs to fdatasync() the journal on commit (which typically issues a whole device cache flush) or use O_DIRECT | O_DSYNC (ending up as a FUA write which just tags the individual write as needing to be durable) for journal writes. Often that drastically increases latency and slows down concurrent non-durable IO, reducing the benefit of deeply queued IO substantially.

On top-line gear this isn't an issue, they don't signal a write cache (by virtue of either having a non-volatile cache or enough of a power reserve to flush the cache). Which then prevents the OS from actually doing more expensive for fdatasync()/O_DSYNC. One also can manually ignore the need for caching by changing /sys/block/nvme*/queue/write_cache to say write through, but that obviously looses guarantees - but can be useful to test on lower end devices.

One consequence of that is that:

> Multiple instances per device are almost a must.

Isn't actually unproblematic in OLTP, because it increases the number of journal writes that need to be flushed. With a single instance group commit can amortize the write cache flush costs much more efficiently than with many concurrent instances all separately doing much smaller group commits.

I guess it still makes sense for higher abstraction levels though, right? Like a filesystem or other shared access to a storage resource. So these asynchronous APIs aren’t writing as directly to storage, they’re placing something in the queue and notifying when that batch is committed.

> Append-only log structures are what NVMe/flash devices live for.

I would think this is also good for filesystems like ZFS, APFS, and BTRFS, yes? I had an inkling but never really looked into it. Aren’t these filesystems somewhat similar to append-only logs of changes, which serialize operations as a single writer?

Reply on news.ycombinator.com