Enhancing our ZFS support on Ubuntu 19.10 | Hacker News Reader

Enhancing our ZFS support on Ubuntu 19.10

ubuntu.com

116 pointsanuragsoni28 comments

Semi-offtopic, but in the article, they link the reason why Ubuntu considers it legal to include binary ZFS:

> The CDDL cannot apply to the Linux kernel because zfs.ko is a self-contained file system module — the kernel itself is quite obviously not a derivative work of this new file system. And zfs.ko, as a self-contained file system module, is clearly not a derivative work of the Linux kernel but rather quite obviously a derivative work of OpenZFS and OpenSolaris.

This makes lots of sense to me as a layperson. Can anyone explain what the counter argument is? If a non-GPL binary zfs.ko isn't allowed, why is a fully-proprietary Celeste.sh (a commercial video game) okay? Because Celeste has fewer privileges?

> why is a fully-proprietary Celeste.sh (a commercial video game) okay? Because Celeste has fewer privileges?

Because Celeste doesn't link into the kernel, there is a system call exemption in the kernel license to make it abundantly clear that user-mode applications are not considered combined or derived works of the kernel.

As far as ZFS.ko is concerned, there's already history of non-GPL filesystem modules being included in the kernel; namely afs. It's the same argument, AFS was designed for another UNIX-like operating system with a similar VFS layer and - rationale that Linus agreed with. It's really other maintainers at this point that are stonewalling ZFS.ko, an issue that will ultimately come down to what a court decides constitutes a derived work.

It depends on the definition of "derived work", which is a legal definition, not a technical definition. For "user programs that use kernel services by normal system calls", everyone agrees that it's not a derived work ("[...] this is merely considered normal use of the kernel, and does not fall under the heading of "derived work"."). For kernel modules, which use the internal kernel API instead of system calls, it's less clear.

Also, you cannot say that zfs.ko "is clearly not a derivative work of the Linux kernel but rather quite obviously a derivative work of OpenZFS and OpenSolaris" - if that were the case, that kernel module clearly wouldn't work, since the internal kernel API of the Linux kernel and of the Solaris kernel is completely different. Whether the changes necessary for zfs.ko to work with the Linux kernel make zfs.ko as a whole a "derived work" of the Linux kernel or not is a more complex question.

But that's not the worst issue. The real problem is: under what license is the whole kernel, once you linked together the common Linux parts (which are under GPL2) and the ZFS parts (which are under the CDDL)? The "mere aggregation" provisions of GPL2 clearly don't apply, and the licenses are incompatible (on purpose, it's been documented that CDDL was intentionally made incompatible with the GPL). Since the CDDL can't apply (due to the GPL parts), and the GPL can't apply (due to the CDDL parts), distributing the combined whole would be a copyright violation.

> But that's not the worst issue. The real problem is: under what license is the whole kernel, once you linked together the common Linux parts (which are under GPL2) and the ZFS parts (which are under the CDDL)? The "mere aggregation" provisions of GPL2 clearly don't apply, and the licenses are incompatible (on purpose, it's been documented that CDDL was intentionally made incompatible with the GPL). Since the CDDL can't apply (due to the GPL parts), and the GPL can't apply (due to the CDDL parts), distributing the combined whole would be a copyright violation.

I asked for the counter argument, so, thank you—but this strikes me as totally silly. If I distribute a zip file with a Linux ISO and an OpenSolaris ISO, is that also a copyright violation?

It seems to me, the Linux kernel and the ZFS module aren't actually being put together until they arrive on the user's machine, and at that point, redistribution is irrelevant.

The CDDL code doesn't depend on Linux internal APIs - it depends on SPL, the Solaris Porting Layer, which implements them using Linux APIs. This way the CDDL code stays independent of linux.

Also, we have one persons word that they were asked to make it incompatible, at the same time we have another people claiming that Sun expected CDDL code to end up in Linux kernel pretty much overnight and was surprised by the reaction, and on the third hand we have developers who campaigned internally at Sun against using GPLv2/v3 (Sun was interested in using GPLv3) because it would prevent integration of opened code with BSDs and the like.

> on purpose, it's been documented that CDDL was intentionally made incompatible with the GPL

Citation needed. It’s actually been documented _by people that were there_ [1] that it was NOT the case that the CDDL was made intentionally incompatible with the GPL.

[1]: https://www.slideshare.net/mobile/bcantrill/fork-yeah-the-ri... (Slide 10) - or in video form with a longer explanation: https://youtu.be/-zRN7XLCRhc

It’s all kind of bogus of course. No one would believe it if Microsoft could somehow claim you need their license to create a driver for Windows and no one would consider it a breach of copyright if the driver uses undocumented API. But somehow for Linux it’s different.

Having been a ZFS fan since the twilight of OpenSolaris, I'm very glad to see ZoL taking off. Rolling it into Ubuntu and making it officially supported was a great move - after some frustration with trying to run ZFS on a CentOS box and having it occasionally break after a kernel update, having it easily available on Ubuntu was like a breath of fresh air.

Having it readily available as a root filesystem, and having TRIM support at long last, is great news.

Happy ZoL user on my home NAS for over 6 years now. 60TB+, not a single byte lost. Very, very happy to see the continued support.

I've been using zfs on Debian for over half a year now. I've been really happy with the stability, going through a handful of power outages and the addition of another drive.

Me too. I’ve been really surprised at how easy it is. Even migrating to a fresh OS was a breeze and didn’t require me backing up any configs.

Do you mean ZFS as root or as a data partition? and when you say migrating to a fresh OS do you mean the Stretch to Buster upgrade? If so, it is great to hear it is working so well now.

Not the grandparent, but I've moved a running system from one ZFS pool to another without physically touching the computer. The OS was running off a ZFS root.

ZFS data volume. Haven't tried root yet (wasn't easily available when I created the volume). Still, a data volume in ZFS is so much better than the traditional md raid. Won't go back.

ZFS lets me back up every `apt-get` action, which is useful on Debian Testing:

    /etc/apt/apt.conf.d/71backup

    DPkg::Pre-Install-Pkgs {"/sbin/zfs snapshot rpool/ROOT/debian@apt-get_$(date '+%Y-%m-%d-%H%M%S')";};

When I inevitably break Debian with no time to troubleshoot, I do

    # zfs list -t snapshot | grep apt\-get
    ...
    # zfs rollback -r rpool/ROOT/debian@apt-get_<insert time stamp here>

I've set up something similar on FreeBSD with just a shell function wrapping pkg and port, very handy.

Nice.

I haven't touched ZFS since it first came out on Solaris around 2008ish (we had a 48 drive "x4500", great hardware, but solaris :/ )

Are the snapshots instant? I'd be tempted to replace "sudo" with something that snapshotted them too!

> Are the snapshots instant?

Yes. But you have to remember to delete the old ones at some point.

> Are the snapshots instant?

Yes, it just copies & stores the block at the top of the filesystem[0], in the same way naming a git tree will keep its content recursively alive.

However all the data in the snapshot is retained as-is until the snapshot is destroyed, so as the system keeps being used and diverges from the snapshot the snapshot is slowly going to "increase in size" compared to the live FS.

[0] there's also a more recent "checkpoint" concept which "saves" the entire pool by copying the uberblock.

The actual act of snapshotting doesn’t do a whole lot. I believe ZFS uses CoW for that purpose. If you modify a file after a snapshot, the file (or block) will be copied first. Normally, with CoW, the original would then be marked as free space available for writing, but if a snapshot exists, it will be preserved. (This is a general concept; I don’t know the nuances of ZFS’ approach.)

I thought that was the case, but in more recent times they only snapshotting I've done is on AWS instances which take forever

Oh my god.

This is such a simple thing, and yet it somehow didn't occur to me until now... if ZFS is your OS's root filesystem, you can use ZFS snapshots to capture and roll back the complete state of your system. Just like that, no other special tools needed. Wow.

This isn't really new. You've been able to do this with LVM for years, but it isn't quite as graceful.

It’s fairly graceful with btrfs, and Ubuntu has been doing that by default if the root fs is btrfs for quite a while. The only trouble is that you’re liable to run out of space in /boot/ eventually.

This is the whole concept of Solaris boot environments

Plugging a small script I wrote that is a bit more targeted than this by only snapshotting the datasets that are going to be changed. Haven't written a readme for it yet (or packaged it :) ) but both files have some header comments explaining how they're used and where they're put.

https://github.com/paxswill/zfs-apt-snapshot

I wonder how's the performance compared to ext4. I've only seen some old articles and not sure if it's comparable to the latest version

Here's a recent test:

https://www.phoronix.com/scan.php?page=article&item=bcachefs...

It needs to be said though, that ZFS requires tweaking for some workloads. In particular, database volumes needs to have the recordsize set to 8 KiB, in order to match the DB page size (I think the benchmark did not do that).

Reply on news.ycombinator.com