Hacker News Reader

Comment by londons_explore | Hacker News Reader

Postgres is copy on write. (A modified record is copied to a new page when written, and the old record left in place till a vacuum).

ZFS is Copy-on-write (One byte written in a block requires the whole block be rewritten, with the old one scheduled for reclamation).

The underlying SSD wear levelling algorithm is Copy on Write (Writing a single byte involves writing the data from the page to a new block, and then erasing the old one sometime later)

That means a tiny 1 byte modification to a postgres record involves creating many new unnecessary copies of a lot of data...

I imagine that if the three layers could be combined, dramatic performance benefits could happen, since written data might go down by an order of magnitude at least....

View on HN

I'm not sure it's an order of magnitude reduction in real flash writes, but there have been some pretty promising results from experiments with collapsing those three layers. Moving it all to the SSD gets you a Key-Value SSD, and moving it all to the host system gets you a Zoned SSD. Both ideas have seen enough interest from storage vendors and hyperscale cloud to end up standardized. So at this point it's mostly a matter of getting support added to common software like Postgres so that it's easy for smaller operations to adopt.

That's what you get when everybody is hiding abstractions in 10 layers.

The same thing happens with all the layers of virtual-machines

To be fair you have some virtualisation stacks (like QEMU/KVM) which goes in the opposite direction:

It provides VMs with drivers for purely virtual “virtio” devices (storage, network, etc) with no effort or overhead put into mimicking the mechanics of a real sas/sata/scsi device or whatever.

The result is less complexity and a much more direct data-path so things should hopefully not only be more performant, but also more stable.

The CPU itself also runs it's own little VM, OS, prediction, caching etc.

Not sure about SSD but filesystem COW doesn't only entail writing and freeing whole blocks, it entails "dont make an actual copy until write"

COW filesystem means you can make (virtual) copies of files /blocks without writing (duplicating) them on the media. They only write metadata for bare copies and delay duplicating data until a virtual duplicate is altered.

Actual writes are written into a free block, then the old block is marked clear. The old block is not copied and then written over. In my understanding that's not what COW characterizes - its refering to how copying data is almost free (only costs metadata changes in COW filesystems) until copies are altered (written to)

These semantics are true at all three layers.

In Postgres, Transactions work on a "snapshot" of the data that existed at one point in time. That snapshot is logically a copy of the data, but in reality uses copy-on-write of records to avoid having to make a copy of the entire database at the start of any transaction.

In ZFS, it works as described.

In SSD's, operating system 'write' commands are treated as transactions - ie. certain ordering semantics must be preserved in case of a power failure. Since performance is improved by having extra parallelism and not doing the actual operations in the order they are presented by the OS, a copy-on-write model is used to ensure that an incomplete transaction can be rolled back. This isn't supposed to be user-visible, but occasionally in a badly broken SSD, you hear users complaining of 'it works fine, but then when I reboot my computer everything I did is undone'! Well that's because no transactions are committing...

What puzzled me is how you feel layers of particularly COW operations combine to make a dramatic increase in write load. I expect the whole block writing requirement of the SSD would be the major and finally unavoidable multiplier of its write load. Unless its blocks are small. There seems to be only a dramatic increase involved if one of the layers has dramatically larger blocks. There would be a middling chance of doubling writes between layers due to block misalignments and overlap, but that will not occur constantly and doubling is the worst case from misalignment. The most significant write multiplication throughout the layers should be the biggest block size as I imagine it.

This is where RocksDB or other SSTable storage systems really line up well with SSDs. The write amplification reduction can improve throughput and disk life by multiples.

https://engineering.fb.com/2016/08/31/core-data/myrocks-a-sp...

> A modified record is copied to a new page when written

AFAIK а modified record is copied to the same page if there is enough space (which you can tune).

Oh, a postgres disk controler / fs / storage engine would be an interesting project.

If anything, it would drastically increase the lifetime of the SSDs.

SSDs already do copy-on-write internally, so this doesn't change much. The files are slightly smaller so it may save a few writes here and there, but I wouldn't expect a drastic change one way or another.

Right, but if it's over-lapping, then wouldn't it move the write around on multiple layers?

wear levelling isn't really copy on write is it?

the original block will be removed from the mapping entirely (not referred to by a copy)

Typically flash devices can't overwrite data directly, and can only erase very large blocks (ie. 1MB or more).

That means any write of data must be made to a new, freshly erased, location.

The old version of the data, sitting in the middle of a large eraseblock is no longer used, but cannot be reused until everything else in the eraseblock is either unused or copied elsewhere.

In the worst case (of a nearly full SSD), it means that for every 1 byte write into a 4k page, a full 1Mbyte of data needs to be copied. Typical cases are better (when the drive has plenty of spare space, so can delay the reclamation of the eraseblock for as long as possible, hoping that other things in the block are invalidated, reducing the amount that needs to be copied).

TL;DR: It's complex, but a lot of copying is involved with most writes...

Reply on news.ycombinator.com