ZFS v0.8.2

There's one command for handling storage pools (zpool, https://manpages.debian.org/unstable/zfsutils-linux/zpool.8....), e.g. adding/replacing disks, monitoring I/O utilization, etc.

And then there's another command for dealing with ZFS datasets (zfs, https://manpages.debian.org/unstable/zfsutils-linux/zfs.8.en...), e.g. setting properties for datasets (quotas, compression, delegation of privileges to non-root users), managing snapshots.

Both CLI commands are scriptable (e.g. -H can be used to suppress human-readable headers and -p turns off "friendly" formatting for numbers) and there are libraries (libzfs, libzpool) which can be used to access their functionality (e.g. for managing snapshots) in your own programs.

I don't think I can do ZFS on Linux (or btrfs for that matter) justice in a (short) comment on HN. There's just so much to it (e.g. SSDs as cache/log devices, sending/receiving snapshots over the network, compression, etc.)

There are some rough edges, of course. Until a few years ago ZFS on Linux had massive problems with resource management, i.e. it would regularly panic in low-memory situations. These appear to be fixed though - at least I haven't seen any panics in the past two or so years.

Due to its license ZFS on Linux will probably never be part of the upstream kernel. We use it on Debian which means we get to compile the ZFS kernel modules on each box when there's a kernel update (with DKMS). Plus there are some pitfalls when using ZFS as your root filesystem (on Debian stretch without systemd services would start before /var/log was mounted).

We've had plenty of disk failures which ZFS detected well before SMART did because we run monthly scrubs for our ZFS pools (i.e. basically ZFS verifies its checksums for all the data that's stored in a pool). Recovery is as easy as popping in a new disk and running "zpool replace".

Ok, this ended up much longer than I had planned. This has to suffice for now.

btrfs uses a more traditional command, subcommand, args setup, which by itself is not a problem, but the actual command and subcommand structure is a dumpster fire to be charitable.

Try managing large groups of snapshots without custom tooling. The whole thing makes me sad really.

btrfs has a lot going for it at the architecture and disk format level (dedup is actually something theoretically useful, while the zfs design was flawed from the start), but the implementation has just never made it.

[0]: https://btrfs.wiki.kernel.org/index.php/RAID56

With the exception of accidentally enabling deduplication on a 16 TB array on a system with 8 GB of RAM, ZFS has been fantastic. Everything works, and works much as you'd expect. The tools and concepts are clear and concise and it's obvious that a lot of work was put into everything from the design to the documentation.

btrfs, on the other hand, has been a nightmare. I set it up on a new desktop machine earlier this week, and I've spent most of this afternoon trying to recover data from it. Due to a power outage, the filesystem is corrupted. I didn't know that right away because I enabled zstd compression in the filesystem but GRUB couldn't boot/read zstd files/filesystems, etc.

A lot of btrfs is counterintuitive, or downright worrying. Most of the threads I've seen regarding filesystem errors end with "I reinstalled and everything is fine". The documentation suggests not trying to `btrfs check --repair`, because apparently that's the wrong way to do things? You should try to mount with the 'recovery' mount flag instead, which is not intuitive.

In my case, the kernel is throwing an error about the checksum map, but it can't rebuild it or repair the filesystem. In a rescue image it errors because it tries to call pthread_cancel but can't load libgcc for whatever reason, so I can't rescue my system from a rescue system.

Even when it did work it was confusing. Unlike ZFS, btrfs subvolumes seem... counterintuitive? On ZFS I can take a recursive snapshot of a volume or subvolume and any subvolumes it has, which allows me to divide a subtree up into multiple subvolumes but still treat it the same (e.g. if I want multiple entries in /var/lib/mysql/<mysql_instance> or something similar). On btrfs, you cannot do recursive snapshots, and I've even seen some people describe this as one of the (few) reasons you'd use subvolumes: to exclude something from snapshots.

After ten years, it feels as though btrfs is 90% done; that is to say, it has 90% of the functionality it should, that those features are 90% done, and that what is done works about 90% of the time.

Honestly, with the state that btrfs is in, I don't understand why it's in the kernel at all, and why distros support it (but I do understand why RHEL pulled it). Working with it directly makes me worry for the data I have stored on my synology, and now I'm wondering if I should have just built a FreeNAS box instead.

TL;DR btrfs is a giant mess and you shouldn't use it. ZFS has more up-front overhead (getting the package versions configured for Ubuntu) and is more difficult to go all-in on (e.g. ZFS root) but at least you won't lose your data out of nowhere.

https://rkeene.org/projects/info/wiki/BtrFS

Overall similar. I ran into more critical bugs with BtrFS, but ZFS (on Solaris) was not perfect (most of our Solaris kernel panics were ZFS-related on Solaris).

I found the same as the comment above, btrfs was a dream to set up and a nightmare when something went wrong, which was several times a year it seemed.

ZFS on the other hand, I had to write scripts to handle my snapshots and snapshot expirations but the filesystem itself has been rock solid.

I haven't put ZFS through it's paces. However with btrfs I had so many issues on a dead simple workload it just didn't seem stable to me. This sucks and I wish I could contribute because I loved their goals and thought they were designing everything right, but stability wins the long game, as it were.

DTrace for example, that stuff is proper. Great to see whats happening with BPF on Linux these days.

I guess it speaks to the engineering efforts behind it all. Not an easy feat to integrate or bolt on. And not to mention the inertia of eco-systems of tooling on a well established platform such as Linux.

Interesting times ahead and great work, my favorite community! :)

This is correct - and good news - but there should be a reality check involved with regard to timelines ...

ZoL code is most likely not going to be part of any 12-RELEASE distributions, which means that:

IF you only run -RELEASE in production (a good policy) AND if you do not put a x.0 release into production (historically, a good policy) then you are 2-3 years away from running this ZoL code:

http://freebsd.1045724.x6.nabble.com/How-many-quot-productio...

As it stands now, our customers want encryption and raw send which we provide to them using ZoL, on 12, in ports form, in a VM.

This is how people zfs-send encrypted snapshots to rsync.net.

Our base ZFS platform, however, will most likely not run a ZFS version that doesn't come with a -RELEASE version of FreeBSD with a x.0 version number. Not a big deal when everyone is using borg and restic anyway, but I digress...

The incident where updating your Ubuntu kernel resulted in data loss probably could not have occurred under the BSD model due to close coupling between the kernel and the filesystem.

I suppose the benefit of using XFS and LVM is that they're both very well understood and widely used technologies. Sometimes incremental evolution is all you need.

I'm following the development with interest, because I think the approach has merits. Dave Chinner's talk about teaching XFS how to do snapshots and subvolumes was particularly interesting. The gist of it, as far as I understand, is that XFS can pretty much do those things already, but it needs better integration for performance and usability.

I have a data store (nightly build archive) that gets almost 3x reduction in space used with dedupe, but the performance hit is just too big for us to use it.