Anyone know how the compression features of bcachefs work? At what level is the compression happening?
So better compression at the cost of slightly reduced performance for random reads.
Edit: I was paraphrasing the text in the peer linked document:
"As with data checksumming, we compress entire extents, not individual disk blocks - this gives us better compression ratios than other filesystems, at the cost of reduced small random read performance."
In practice being a true extent-based filesystem with compression and snapshotting is one of bcachefs's distinguishing features (snapshotting is rather obvious in a block-based system by comparison). Other systems work similarly at a high level but the details are of course quite different in practice.
How do bcachefs' extents differ from ZFS's logical blocks (whose size are controlled by the dataset's recordsize property) and how do bcachefs' blocks differ from ZFS's physical blocks (whose size granularity are controlled by the ZFS pool's ashift property)?
In contrast, ZFS logical blocks have a fixed size, but may be handled "sparsely" by an underlying number of (smaller) physical blocks. So a 128KiB logical block doesn't mean every 4KiB file takes up 128KiB, naturally. But to change 'recordsize' in ZFS, you also need to rewrite the file, since the logical block size won't be retroactively changed. For good reason, of course. But the indirection between physical/logical makes a big difference in the underlying mechanics.
Snapshots make a lot of sense in a system that uses logical, fixed-sized blocks as the atomic "unit" of data even if there is an underlying physical block layer. For example, COW is trivial -- page is read, written and marked dirty, then written out to a new copy of the block it came from. The original logical block can be left alone, and the cost is understood as proportional to the underlying number of physical blocks rewritten (and some change to update metadata). But how does COW or snapshotting work in an extent system? In a non-COW system, you can update the data inside the extent in-place since you don't copy, but in a COW system you need to copy or potentially split the extent which could be costly. You may actually want to group extents together if they're for multiple versions of a file when doing COW. There is no layer of "indirection" above this, so that's where a lot of the trickiness comes from, I think. If you have indirect "logical pointers" instead, you can defer a lot of work, things like reclamation, fragmentation, splitting and merging, etc.
There are some other interesting design points in the notes[1] I was looking over to refresh myself, but I don't know how some of them relate to ZFS specifically. For example, another thing bcachefs is pretty good at is low tail latency, and the note on eschewing heavyweight transactions/locks is really good example of that. If you want to run 'touch file', you don't need to take a transaction lock, create an inode, then create the dirent, and release the lock. You can instead simply create the inode and then create dirent assuming that they have a proper write barrier between them, and deleting a file is the inverse operation. Then the system can never be left in an inconsistent state, it can only result in a few dangling objects in the key-value store, which can be garbage collected later.
The low tail latency is also assisted by having really small amounts of metadata for each inode, just barely a 64-byte cache line IIRC; so some really weird pathological cases like `ls` in a directory with 500k dirents is way faster in bcachefs than elsewhere. I also wonder if that also falls out from some of its basic design; it basically looks like a POSIX filesystem layered on top of a fast, ordered b-tree, so there's the storage layer and the actual filesystem semantics on top. The filesystem layer probably doesn't actually need to record that much in-band metadata for a given inode when a lot of the underlying consistency and performance issues are handled elsewhere. But that's just me thinking out loud.
Overall, there are lots of differences in the details even if some of the high-level stuff comes across as the same.
A ZFS physical block also records a variable, contiguous range of bytes on-disk, not unlike what you describe with bcachefs extents (although, in ZFS the "contiguous" part can get more complicated due to gang blocks and RAID-Z, I think).
That said, if you don't have compression enabled, then in ZFS the physical size of the block usually does correspond to the logical size, regardless of how much data was written to it (except for files smaller than one logical block size).
Also, if I remember correctly, a ZFS logical block usually only maps to one ZFS physical block, although it can map to up to 3 physical blocks, each of them having the same data (the actual number depends on how many redundant physical copies the logical block is configured to have on top of the ZFS pool redundancy).
But in any case, thank you for your long explanation. Some details still escape me, but I can consult the architecture document you linked if I have more doubts.
So a ZFS block is even larger than a bcachefs "extent".
As an example, in terms of actual on-disk storage, a 128 KB logical block in ZFS can be compressed down to (for example) a 3.5 KB "physical" block and it will only actually consume 3.5 KB of disk space (surprisingly, the block size of a ZFS physical block doesn't have to be a power of 2!).
That said, if you have 4 KB-sector disks then you should probably use a physical block size granularity of 4 KB for your ZFS pool rather than 512 bytes.
So, regardless of the file size, an on-disk ZFS block can actually occupy anywhere from 0 bytes up to 1 MB, with a granularity of up to 512 bytes, depending on how much it is compressed, what the physical block size granularity of the ZFS pool is and what you've selected as the recordsize property (i.e. the logical block size) for the corresponding filesystem in the pool.
According to Allan Jude (OpenZFS & FreeBSD developer), you should never use 512-byte recordsize even on an HDD with 512-byte sectors. Always start from 4 KiBs, probably even from 8.
https://klarasystems.com/articles/tuning-recordsize-in-openz...
I think you meant to refer to the pool's sector size (i.e. "ashift" property) rather than the recordsize.
I agree, but I wouldn't say "never", as that's too strong a word.
It would be useful to use a 512-byte ZFS pool sector size even when the HDD uses 4K sectors if you care a lot about disk space efficiency but don't care at all about performance or HDD wear (for example, if you plan to use a ZFS pool almost exclusively read-only).
That said, I would also recommend a 4K pool sector size in almost all cases, as HDD space is cheap.
I think that post is warning you that it's immutable once done. If you want to be more flexible in the future and can't be sure you won't need to use 4kn/512e disks at some point, yes, use 4kn (or with all SSD storage it gets even messier).
But using 512b can be a significant gain on space efficiency for some people.
I think we're talking past each other. In a practical sense I agree the two filesystems achieve an essentially identical result through very similar methods, as you have said.
I am making a point specifically about the details of the implementation logic of the two filesystems. Bcachefs (as I understand it) combines multiple smaller blocks into a larger extent. ZFS only has one raw record and its corresponding on-disk compressed record, whose size is rounded up to ashift.
The default was to limit it to 1M in case there were weird edge cases with the larger sizes (16M is the largest it'll encode on disk without a format change, IIRC), with larger sizes behind a tunable, but after a few years of people who knew this twiddling it, the default in 2.2 will be that the tunable is set to 16M.