The potential risk to ZFS created by the shift in its userbase
utcc.utoronto.ca
utcc.utoronto.ca
Since FreeBSD support is going to be merged directly into the ZoL repo, ZoL devs will see the FreeBSD CI build results and it will be a lot closer to a unified upstream than the past. Being a downstream instead of integrated into the source of truth made feature development a laborious two way sync and communications were spread over many mailing lists and bug trackers. I think this move will be good for both communities, companies like iX Systems will help get trim across the finish line in ZoL.
In terms of shifting userbase, FreeBSD might even win in terms of ZFS revenue or other metrics like shipping systems due to FreeNAS (would have to spend a few hours researching the ZoL companies business models since a lot of the work is subsidized by the US Government via LLNL). I don't think there will be any tectonic shifts in userbase, there are still pros and cons to both Linux and FreeBSD in different areas and being behind in ZFS wont be wont be a con for either now.
Unrelated to this post, I'm quite interested in knowing how Canonical's move has worked out so far. Does anyone know if they've got customers at any meaningful scale who are using ZoL ? Or is Canonical contributing to ZoL ? There may not be an immediate shift in userbase, but Canonical can certainly make a more comprehensive bid to future customers with their overall offering that also includes ZoL, compared to a specialty storage provider that supports ZFS on FreeBSD.
ZFS on Linux is loaded via DKMS. DKMS is basically the compile-from-source alternative to binary blob drivers. Is there a reason to believe that ZFS via DKMS is materially different, legally, to Nvidia drivers being loaded via binary modules?
Indeed it appears that Nvidia proprietary drivers these days go via the DKMS route. Are they also at risk?
If ZFS was a module within the Linux kernel source tree, Linux kernel devs would keep it in mind when making changes to the Linux kernel's APIs. They wouldn't make potentially breaking changes to that API, or replace APIs with ones that can't be made compatible with ZFS at all, without considering the effect it would have on ZFS. The Kernel devs have to take the whole tree into account, including all the modules.
But since ZoL is not part of the kernel, and is instead a separate independent module that gets loaded into the kernel, the ZoL have to play catch up. Instead of Kernel devs working with them to make sure things stay compatible, or giving them a heads up when something needs to be changed for a new version, the ZoL team is on the outside, trying to make sure they don't get boxed out of the Kernel API.
It can be a tenuous position to be in. For a more everyday example, look at the Firefox and Chromium browser updates. For Firefox Quantum they completely revamped their extensions system, making all existing extensions worthless. Either the extension devs rewrote them with the new API, or the extension is just gone. Similarly Chromium is changing its API for how extensions can interact with the DOM and networking, which would make Ublock unusable. If these features were officially part of the browser or official extensions, the browser devs would work with the extension devs to make sure things stayed working; instead the extension developers need to keep up or watch their product die.
If the Linux Kernel completely changed its module API, and ZoL was caught flat footed and needed to make substantial rewrites, what would the projects backers do? Would they commit the dev time and resources needed to make those changes, or would they compare the benefits of ZFS vs the competition -- solutions based on btrfs or LVM, both built into the kernel -- and decide to jump ship before committing more time? That's where the danger really lies. If the Linux Kernel makes a serious breaking change, ZoL's supporters might double down to fix it, or they might just cut their losses.
And while there is an official way to integrate external filesystems into the Linux kernel, FUSE, it's extremely slow compared to Kernel native modules. There was a userland FUSE implementation of ZFS, and there's a reason it's fallen out of favor. ZoL is significantly faster because of its direct kernel extension, but it's playing with fire by not being part of the kernel tree itself.
Which would essentially be impossible.
OpenZFS is designed in a way that uses a shim layer between ZFS's internal Solarisish API usage and the native OS's usage This allows the same ZFS code to run essentially unmodified on many *nixes without major changes.
In ZoL, this is called SPL (Solaris Porting Layer), and is one of the two kernel modules required to use ZoL. Linux does not export the proper APIs that ZFS requires to work here, and SPL fills that gap.
ZFS modules do "break" often due to internal API changes, but the fix is usually shipped in ZFS stable before the kernel itself is shipped. Only people who follow ZoL development closely ever see the sausage being made.
It is highly unlikely that Linux can ever do anything that ends up with a situation that the SPL and ZFS kernel modules can't easily #ifdef their way out of. It wouldn't make sense for Linux to, either.
Eh, it depends on the distribution. I use Fedora, and there have been quite a few times when a kernel update has resulted in ZoL becoming useless for a while until they push an updated version. The solution, of course, is just booting the earlier kernel until ZoL catches up.
In fact, this is happening right now. The 4.20 kernel was pushed to updates on Fedora (at least my system) on 1/23. The latest stable ZoL doesn't work with it. They're on RC3 for the next release, though, and that does. Hopefully it comes out soon.
In the mean time, I'm running the last 4.19 kernel. shrug
Alternatively, you can use the variation of a distro, if it exists, that provides a more stable kernel and package version set. In this case, that would be RHEL/CentOS.
For a workstation, that can be annoying, but as you said, just use an older kernel for a bit longer. Perhaps mark it as to be skipping in normal package operations, and have a cron job that runs to check the kernel specifically and emails you if there are updated versions of the kernel that exist.
For a server, I imagine it's rarely a problem since those should be running more stable distros anyway, since 99% of the time the kernel is older and back-patched (it may be the current kernel, but still weeks after it was released right at point release updates), which should result in a stable API for ZoL.
The trade-offs there don't seem too onerous to me. A little hand configuration for a workstation (of which there's likely one or two for a person to deal with as long as there's smooth server support which is generally both more important to have solid because it can be harder to fix if there's a problem and because it can scale from none to many per person.
If a security fix breaks your ZoL integration, my guess is you're actually better off waiting for that to play out and resolve itself than to expect it to work. If a feature back port breaks it, that might be a little more annoying, but I imagine it will be fixed in short order, and you only have to worry about that once every 6-12 months (and it's well publicized).
> Those kernel updates once broke HP b120i properietary driver (HP releases new version of this driver for every minor RHEL release).
If HP is releasing closed source drivers for RHEL, I imagine they would want to be on the certified hardware list and test, or at a minimum seek access to the beta spins of the point releases (which I think is where it broke) so they can test before it comes out. I'm not sure I blame Redhat for HP trying to specifically support RHEL and failing to do so, given the systems I know they have in place to help companies in just that situation (because it helps RHEL users).
In any case, all I'm really noting is that between Fedora, which ships a new kernel version every kernel update (AFAIK) and RHEL/CentOS, which ship larges the same kernel with only the specific changes needed the majority of the time, keeping ZoL working should be vastly easier on RHEL systems (and in fact, any OS which does back patching of kernels, which I believe includes SuSE and the LTS releases of Ubuntu).
1: https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#RHEL_...
And the Linux kernel has made numerous breaking changes to their APIs that ZoL has been able to work around. So it has happened, they've just been able to deal with it.
Despite the belief that a breaking change that ZoL can't work around being improbable, it is still possible. The Linux Kernel could majorly overhaul an API in a major version release, in a way that ZoL can't handle. Given ZoL's status as a separate kernel module not under a GPL license, it's entirely possible that no amount of yelling gets the Linux maintainers to change their mind. In fact Wowfunhappy notes a discussion along those lines is happening currently based on function symbols being removed for an API required for Mac hardware support.
And sure, that compatibility layer means that users could switch to FreeBSD, or Solaris, etc, and keep using ZFS. If that happens, does Delphix change their target platform again to move to FreeBSD? Or do they come up with another solution, and stop supporting ZoL?
The last paragraph is pretty telling too.
> My tolerance for ZFS is pretty non-existant. Sun explicitly did not want their code to work on Linux, so why would we do extra work to get their code to work properly?
Ouch. It hurts to see Greg still holding onto that old grudge.
Wariness around commercial Unix vendors may've made some sense in 2005 when ZFS was released, but not only does it not make sense 14 years later, but the company the community viewed with suspicion has since entirely ceased to exist.
I spent a long time trying to finagle btrfs because it was the blessed, in-tree copy-on-write filesystem. It was a nightmare. It doesn't take long for ZFS to prove itself the massively superior solution.
Canonical's adoption of ZFS is a welcome relief, along with SFLC guidance that there is no incompatibility with the CDDL and GPL.
We need to bring the rest of the crew along and stop reinventing the wheel here. Linux will be so much better off once it accepts ZFS.
Its why I've never been a fan of the GPL. Its restrictive instead of permissive.
I've been using Ubuntu since 8.04, ran a LoCo team and even did some package maintenance, and I've watched Ubuntu embrace tech whole heartedly, and then drop it like a rock 2 releases later, over and over again. Ubuntu moves fast and breaks things, and them promoting something should not indicate to you that they'll promote it for the long term. Long Term Support does not mean long term new development, and LTS support does not mean helping to implement support in new major Kernel versions.
1. https://blog.ubuntu.com/2018/01/26/bionic-beaver-18-04-lts-t...
2. https://arstechnica.com/information-technology/2018/05/ubunt...
I'd agree their support of ZFS seems to be in odd state, Neil McGovern had to deal with that weirdness as DPL [1] which was very annoying as the DKMS was a "good enough" solution.
[1] https://go.pardot.com/l/6342/2017-10-24/3xr3f2/6342/188781/P...
But... that was already the case.
> Yeah, HELL NO!
Guess what? You're wrong. YOU ARE MISSING THE #1 KERNEL RULE.
We do not regress, and we do not regress exactly because your are 100% wrong.
And the reason you state for your opinion is in fact exactly WHY you are wrong.
Your "good reasons" are pure and utter garbage.
The whole point of "we do not regress" is so that people can upgrade the kernel and never have to worry about it.
-Linus Torvalds
Specifically, your example brings up the danger that ZoL is in. The Linux kernel does everything in their power to not break User Land APIs. ZoL isn't User Land, and it replaced the previously extremely slow User Land ZFS Linux implementation. It is a Linux kernel module, and as [2] notes, Linux kernel devs are absolutely allowed to make breaking changes to internal APIs. The cost of the increased speed of a Kernel module is the risk of internal API breaking changes. I highly doubt that Linus would scream anywhere near as much about an internal API change that broke a separately maintained linux module with non-GPL code.
1. https://stackoverflow.com/questions/24897511/what-is-the-rep...
It seems like an exactly parallel situation to me.
(PS: further, I don't think you can separate out the legal risk from attitude / API risk, because of things like this, mentioned elsewhere in this conversation: https://marc.info/?l=linux-kernel&m=154722999728768&w=2 - specifically marking symbols GPL to make them non-available.)
But the real price of being an external module is very clear for Nvidia proprietary driver issues. If you look at distro and kernel bug reports, you'll see piles of reports for Nvidia driver users, which devs won't look into because of the tainted kernel. Some Linux software even blacklists Nvidia driver users from using graphics hardware acceleration for their apps, because it causes so many bugs. Nvidia is operating outside of the open source ecosystem, which means they don't benefit from that ecosystem like open source implementations do. When an Nvidia driver user runs into a bug, they're usually just told to shove off and not use the driver.
I've had the DKMS modules break on what were supposed to be minor API compliant kernel upgrades on a major distro. The code was then not fixed for a month (our fix was to rollback kernel versions and just tag it there, it's since been removed entirely).
Soured me on using ZFS for Linux at all in production.
As a reminder, those bus errors are supposed to happen when a client-side mapped file has been unlinked on the server. So the ZFS export does not provide pages to the client on a client page fault the same way NFS would on deletion. Tried dedup and compression on and off, no change. A reboot of the server clears it up for a couple of days, but then it starts the same way. Something seems very broken in caching there.
This popped up during maybe the last 4-5 months.
You have a policy that you can't break users' programs, but borking their whole file-system is fine? Keep feeding end-users fears of open source petty feuds making people miserable. That works.
If ZFS wanted to live in userspace (and get Linux's userspace API guarantees) then they could have made a FUSE filesystem instead.
SPL now handles the guarantee to separate ZFS's userland components from Linux's constant internal kernelspace API churn. ZFS, internally, doesn't really know what a Linux is beyond a few little bits here and there.
These changes were made years later, but the developer of ZFS-FUSE disappeared, so they were never used.
The FUSE v3 API further builds on that, and today, it's definitely possible to build a high-performance filesystem using FUSE. In fact, there are many examples of this in the wild: GlusterFS, Ceph, NTFS-3G, and so on.
Today, there's absolutely _no_ reason ZFS would not sensibly work as high performance filesystem entirely from userspace. In fact, it still remains an open issue and a desirable target for ZoL: https://github.com/zfsonlinux/zfs/issues/8
I'll admit, I'm all talk and if it really mattered, pull requests are probably welcome.
End-user dont'care about splitting hairs. Iwas using Linux in the mid-90's and that kind of stance is what turned me off. The sound card would brek, the networking would break, configuring the modem would be hell after each upgrade.
The attitude that you must not break LibreOffice is not okey but breaking the file-system okay 'because it's a driver!' is just untenable.
Making people's computer fail will turn users away. It's not just a Linux thing. When Windows breaks stuff, all hell break loose on the Internet. The problem with OSS, is that its maintainers have no direct, immediate monetary incentive to amend. The long-term loss of confidence is hard to measure, but it's real.
Any reason why?
"My tolerance for ZFS is pretty non-existant" Greg Kroah-Hartman
"please switch to FreeBSD instead of advocating to violate the copyright and licensing rule on my and others work." Christoph Hellwig
Which is a shame, because it would have been nice to see if the shim usage pattern is actually a violation of the GPL - a lot of us would like a clear ruling there. I'm sympathetic to his viewpoint, but that whole ordeal was a bit of a "wtf?" because of how it was handled, and I can't imagine him being sympathetic to anything else in an even somewhat similar situation.
But then why allow them?
I don't get it -- do they care more about what the users want, or about enforcing copyleft? If it's the latter, then they should be happy if those users who don't care about the GPL move to BSD. If it's the former then they should stop playing games and implement a fair policy for all non-GPL module authors.
Its not non-GPL. Its non-GPl-compatible.
BTRFS is basically on maintenance mode, and no one at Redhat/IBM or Ubuntu has any interest in keeping it alive. Anyone that requires HA services will not be using BTRFS.
Redhat ships with XFS as the default file system, Ubuntu ships with Ext4 as default with heavy install-time ZFS support for enterprise storage, and BTRFS is no longer being spoken of by anyone as a potential Ext5 candidate.
Btrfs is on maintenance mode? Based on what evidence? There's 1k-2k commits per kernel cycle. There are dozens of developers actively working on it. Facebook is certainly using it in production for high availability in concert with Gluster FS, they're also using Btrfs almost exclusively with Tupperware, their containerization service.
Red Hat has quite a lot of device mapper, LVM, and XFS developers so it makes sense for them to work on storage solutions that continue to build on that. And Red Hat hasn't been significantly contributing to Btrfs for many years now, so their lack of interest isn't new and hasn't affected Btrfs development.
You've never seen the same team working on two very separate code bases or products?
I have no idea if the btrfs and zfs teams at Oracle were merged, but I don't know that "they are two separate products" is actually a real argument that they weren't. Product teams working on separate things get merged all the time.
You can track developers through their git commits, and you'll see even when they change companies, they almost invariably keep working on the same filesystem they were before the move. Which is why the idea that Red Hat has no one working on Btrfs, means they want to see it go away is not what's going on at all. They have a lot of other developers already, who'd lose years becoming familiar with Btrfs (or ZFS for that matter). So you're going to see them build on what they already are familiar with, rather than moving laterally to a filesystem technology they're not familiar with.
However, I'm pretty sure if I boiled the commit history down for the kernel, most of Facebook's commits would be Flashcache, and not BTRFS.
What I forgot, though, is SuSE supports it as their enterprise FS of choice, which is maybe more important than Facebook.
If you work some place that doesn't have that kind of luxury, perhaps a little extra conservatism is warranted before you go deploying across "millions of servers".
I can't even imagine the hassle of backporting stuff from much newer kernels into 3.10 to make btrfs work.
If you want to use btrfs in production (not the best idea, IMHO, but orgs such as facebook do it) you absolutely need to be running a 4.9.x or better kernel.
It might be a brilliant file system, but I'm never going ot use it because its such an arse to learn how to use it properly.
https://docs.oracle.com/cd/E52668_01/E54669/html/ol7-about-b...
However, I often end up at the Arch Linux wiki, too:
At the end of the day btrfs and ZFS both are just ways to help single-system administrators do better and more reliable backups. I know that's not how they're positioned, but really that's all they do.
ZFS on my laptop took me an evening to setup, but it's been completely hassle free. ZFS is really easy to use.
ZFS in particular seems to push more toward RAIDZ, whereas I'm a bit averse to striping in general nowadays for data recovery/reliability reasons (RAID1 + JBOD is dead-simple to recover, more flexible with heterogeneous disk sizes, and much easier to reason about IMO).
Having snapshots has saved me occasionally and find-new is really useful.
Of course ZFS has a lot more potential to be broken by kernel updates than Java, but the parallel is interesting.
And it's not as if that admission was needed, as anything else would have been insane from Sun.
They were losing badly to Linux, so in the eleventh hour they decided to go open source in an attempt at getting more mindshare = marketshare.
Solaris had some real technical advantages over Linux like ZFS and DTrace, however if they licensed Solaris with a GPLv2 compatible license, Linux could just incorporate their coveted tech, which would be business suicide. Hence CDDL.
You can't blame them given the situation they were in.
That'd mean Oracle actually using their army of litigators for good, though, and it'll be a cold day in Hell when that happens.
What you want is to store the tag metadata heap in an easily indexed structure (i.e: a search tree.) That heap then just contains pointers into content addressable storage. You can kind of build such a thing on top of existing filesystems which support hardlinks, however I'd advise against that for the metadata itself. Once you have the power of tagged content a natural evolution is querying that data using set operators. You will inevitably re-invent (some approximation of) SQL once you arrive at this discovery.
On a NVMe drive rebuilding the search index shouldn't be too costly, even on hundred thousands of files.
I think there were also some ioctl additions for btrfs that let you sequentially walk the filesystem by extents or inodes instead of by directory tree, which makes indexing a lot easier.
But not tags themselves- so I'll still like a file-system that supports tags natively.