Linus: Don't Use ZFS
realworldtech.com
realworldtech.com
"honestly, there is no way I can merge any of the ZFS efforts until I get an official letter from Oracle that is signed by their main legal counsel or preferably by Larry Ellison himself that says that yes, it's ok to do so and treat the end result as GPL'd.
Other people think it can be ok to merge ZFS code into the kernel and that the module interface makes it ok, and that's their decision. But considering Oracle's litigious nature, and the questions over licensing, there's no way I can feel safe in ever doing so.
And I'm not at all interested in some "ZFS shim layer" thing either that some people seem to think would isolate the two projects. That adds no value to our side, and given Oracle's interface copyright suits (see Java), I don't think it's any real licensing win either."
I understand Linus reasoning but there is just no way I will install btrfs, like ever. I rather dont update kernel (I am having zfs on fedora root with degular kernel updates and scripts which verify that everything is with kernel modules prior to reboot) than use file system that crashed twice in two years.
Yes it is very annoying if update crashes fs, but currently:
- in 2 years two time btrfs crashed itself
- in next 2 years update never broke zfs
As far as I am concerned, the case for zfs is clear.
This might be helpful to someone: https://www.csparks.com/BootFedoraZFS/index.md
Anyway Linus is going too far with his GPL agenda, the MODUL_LICENSE writting kernel moduls explains why the hardware is less supported on linux - instead of devs. focusing on more support from 3rd party companies, they try to force them to do GPL. Once you set MODUL_LICENSE to non GPL, you quickly figure out that you can't use most of kernel calls. Not the code. Calls.
Relaxing on anything more permissive than GPL2 would instead mean the end of Linux as we know it. A more permissive license means that nothing would prevent Google or Microsoft from releasing their own closed-source Linux, or replacing the source code of most of the modules with hex bloats.
I believe that GPL2 is a good trade-off for a project like Linux, and it's good that we don't compromise on anything less than that.
Even though I agree on the superiority of ZFS for many applications, I think that the blame for the missed inclusion in the kernel is on Oracle's side. The lesson learned from NTFS should be that if a filesystem is good and people want to use it, then you should make sure that the drivers for that filesystem are as widely available as possible. If you don't do it, then someone sooner or later will reverse engineer the filesystem anyway. The success of a filesystem is measured by the number of servers that use it, not by the amount of money that you can make out of it. For once Oracle should act more like a tech company and less like a legal firm specialised in patent exploitation.
> or replacing the source code of most of the modules with hex bloats.
Ok good point, I am no longer pissed off on MODULE_LICENSE, didn't even thought about that.
I did use it as well many years ago (probably around 2012-2015) in a raid5-configuration after reading a lot of positive comments about this next-gen fs => after a few weeks my raid started falling apart (while performing normal operations!) as I got all kind of weird problems => my conclusion was that the raid was corrupt and it couldn't be fixed => no big problem as I did have a backup, but that definitely ruined my initial BTRFS-experience. During those times even if the fs was new and even if there were warnings about it (being new), everybody was very optimistic/positive about it but in my case that experiment has been a desaster.
That event held me back until today from trying to use it again. I admit that today it might be a lot better than in the past but as people have already been in the past positive about it (but then in my case it broke) it's difficult for me now to say "aha - now the general positive opinion is probably more realistic then in the past", due e.g. to that bug that can potentially still destroy a raid (the "write hole"-bug): personally I think that if BTRFS still makes that raid-functionality available while it has such a big bug while at the same time advertising it as a great feature of the fs, the "irrealistically positive"-behaviour is still present, therefore I still cannot trust it. Additionally that bug being open since forever makes me think that it's really hard to fix, which in turn makes me think that the foundation and/or code of BTRFS is bad (which is the reason why that bug cannot be fixed quickly) and that therefore potentially in the future some even more complicated bugs might show up.
Concerning alternatives:
I am writing and testing since a looong time a program which ends up creating a big database (using "Yandex Clickhouse" for the main DB) distributed on multiple hosts where each one uses multiple HDDs to save the data and that at the same time is able to fight against potential "bitrot" ( https://en.wikipedia.org/wiki/Data_degradation ) without having to resync the whole local storage each time that a byte on some HDD lost its value. Excluding BTRFS, the only other candidate that I found is ZFSoL that perform checksums on data (both XFS and NILFS2 do checksums but only on metadata).
Excluding BTRFS because of the reasons mentioned above, I was left only with ZFS.
I'm now using ZFSoL since a couple of months and so far everything went very well (a bit difficult to understand & deal with at the beginning, but extremely flexible) and performance is as well good (but to be fair that's easy in combination with the Clickhouse DB, as the DB itself writes data already in a CoW-way, therefore blocks of a table stored on ZFS are always very likely to be contiguous).
On one hand, technically, now I'm happy. On the other hand I do admit that the problems about licensing and the non-integration of ZFSoL in the kernel do have risks. Unluckily I just don't see any alternative.
I do donate monthly something to https://www.patreon.com/bcachefs but I don't have high hopes - not much happening and BCACHE (even if currently integrated in the kernel) hasn't been in my experience very good (https://github.com/akiradeveloper/dm-writeboost worked A LOT better, but I'm not using it anymore as I don't have a usecase for it anymore, and it was a risk as well as not yet included in the kernel) therefore BCACHEFS might end up being the same.
Bah :(
When it comes to predicting your future, though, your personal anecdotes may not hold water to more substantial data.
As far as it needing a lot of memory, that is also not true. The ARC will use your memory if it's available, because it's available! You paid good money for it, so why not actually use it to make things faster?
- The kernel team may break it at any time, and won't care if they do.
- It doesn't seem to be well-maintained.
- Performance is not that great compared to the alternatives.
- Using it opens you up to the threat of lawsuits from Oracle. Given history, this is a real threat. (This is one that should be high for Linus but not for me - there is no conceivable reason that Oracle would want to threaten me with a lawsuit.)
> It doesn't seem to be well-maintained.
The last commit is from 3 hours ago: https://github.com/zfsonlinux/zfs/commits/master. They have dozens of commits per month. The last minor release, 0.8, brought significant improvements (my favorite: FS-level encryption).
Or maybe this is referred to the 5.0 kernel (initial) incompatibility? That wasn't the ZFS dev team's fault.
> Performance is not that great compared to the alternatives.
There are no (stable) alternatives. BTRFS certainly not, as it's "under heavy development"¹ (since... forever).
> The kernel team may break it at any time, and won't care if they do.
That's true, however, the amount is breakage is no different from any other out-of-tree module, and it unlikely to happen with a patch version of a working kernel (in fact, it happen with the 5.0 release).
> Using it opens you up to the threat of lawsuits from Oracle. Given history, this is a real threat. (This is one that should be high for Linus but not for me - there is no conceivable reason that Oracle would want to threaten me with a lawsuit.)
"Using" it won't open to lawsuits; ZFS has a CDDL license, which is a free and open-source software license.
The problem is (taking Ubuntu as representative) shipping the compiled module along with the kernel, which is an entirely different matter.
---
[¹] https://btrfs.wiki.kernel.org/index.php/Main_Page#Stability_...
I don't think it has to be conceivable with Oracle...
Unfortunately I have to agree with Linus on this one. Messing with Oracle's stuff is dangerous if you can't afford a comparable legal team.
Money. Anecdotally that's the primary reason Oracle do anything.
Don't be so sure about this.
CoW filesystems do trade performance for data safety. Or did you mean there are other _stable/production_ CoW filesystems with better performance? If so, please do point them out!
No. Distributing (ie. precompiled distro with ZFS) will. You are free to run any software on your machine as you so desire.
Linux project maintains compatibility with userspace software but it does not maintain compatibility with 3rd party modules and for a good reason.
Since modules have access to any internal kernel API it is not possible to change anything within kernel without considering 3rd party code, if you want to keep that code working.
For this reason the decision was made that if you want your module to work you need to make it part of Linux kernel and then if anybody refactors anything they need to consider modules they would be affecting by the change.
Not allowing the module to be part of the kernel is a disservice to your user base. While there are modules like that that are maintained moderately successfully (Nvidia, vmware, etc.) this is all at the cost of the user and userspace maintainers who have to deal with it.
Use FreeBSD where there's a stable ABI and you don't have these problems.
That's going to be plenty of reason not to use ZFS for most people. The licensing by itself is also certainly a showstopper for many.
But I'm not sure his other comments are really fair and, had Oracle relicensed ZFS n years back, ZFS would almost certainly be shipping with Linux, whether or not as the typical default I can't say. It certainly wasn't just a buzzword and there were a number of interesting aspects to its approach.
> It was always more of a buzzword than anything else, I feel, and the licensing issues just make it a non-starter for me.
So presumably the licensing problem mentioned by your parent's comment is weighing heavily here. I think this "don't use ZFS" statement is most accurately targeted at distro maintainers. Anyone not actually redistributing Linux and ZFS in a way that would (maybe) violate the GPL is not at any risk. That means even large enterprises can get away with using ZoL.
See also https://www.kernel.org/doc/html/latest/process/stable-api-no...
(fuse is a stable user-space API if you want one ... it won't have the same performance and capabilities of course ...)
Maybe, but the complains seem to be more related to the (problematic) changes not being of technical nature accidentally braking ZFS, but being more of political nature. With speculation that it might have been meant to _intentionally_ brake ZFS and then pretend this was a accident because ZFS isn't (and can never) be maintained in tree. Basically on the line of "we don't like out of tree kernel modules so we make the live hard for them". No idea if this is actually the case or people just spin thinks together. Even if it is the case I'm not sure what I should think about, because it's at least partially somewhat understandably.
This has come up many times in the past. Keep in mind that linux has always been GPLv2-only, it is not LGPL or anything like that.
https://lwn.net/Articles/769471/
When he says that, I think on the $500 million Sun spent on advertising java.
> as far as I can tell, it has no real maintenance behind it either any more
Which simply isn't true. They just released a new ZFS version with encryption built in (no more ZFS + LUKS) and they removed the SPL dependency (which didn't support Linux 5.0+ anyway).
I use ZFS on my Linux machines for my storage and I've been rather happy with it.
"Don't use ZFS. It's that simple. It was always more of a buzzword than anything else, I feel, and the licensing issues just make it a non-starter for me.
The benchmarks I've seen do not make ZFS look all that great. And as far as I can tell, it has no real maintenance behind it either any more, so from a long-term stability standpoint, why would you ever want to use it in the first place?"
The thing about ZFS that actually appeals to me is how much error-checking it does. Checksums/hashes are kept of both data and metadata, and those checksums are regularly checked to detect and fix corruption. As far as I know it (and filesystems with similar architectures) are the only ones that can actually protect against bit rot.
https://github.com/zfsonlinux/zfs/wiki/Checksums
> And as far as I can tell, it has no real maintenance behind it either any more, so from a long-term stability standpoint, why would you ever want to use it in the first place?"
It has as much maintenance as any open source project: http://open-zfs.org/. IIRC, it has more development momentum behind it than the competing btrfs project.
From my perspective, it has no real competitor under linux, which is why I use it. I don't consider brtfs mature enough for critical data. (Others can reasonably disagree, I have intentionally high standards for data durability.)
Aside from legal issues, he's talking out of his ass.
If there is no "approved" method for creating Linux drivers under licenses other than the GPL, that seems like a major problem that Linux should be working to address.
Expecting all Linux drivers to be GPL-licensed is unrealistic and just leads to crappy user experiences. nVidia is never going to release full-featured GPL'd drivers, and even corporative vendors sometimes have NDAs which preclude releasing open source drivers.
Linux is able to run proprietary userspace software. Even most open source zealots agree that this is necessary. Why are all drivers expected to use the GPL?
---
P.S. Never mind the fact that ZFS is open source, just not GPL compatible.
P.P.S. There's a lot of technical underpinnings here that I'll readily admit I don't understand. If I speak out of ignorance, please feel free to correct me.
There is little incentive for Nvidia to maintain a linux specific driver, but because it is closed source the community cannot improve/fix it.
> Why are all drivers expected to use the GPL?
I think the answer to this is: drivers are expect to use the GPL if they want to be mainlined and maintained - as Linus said: other than that you are "on your own".
I think parent comment wasn't asking for third party, non-GPL drivers to be mainlined, but for a stable interface for out-of-tree drivers.
The idea that Linux needs better support for out of tree drivers is like someone going to church and saying to the priest "I don't care about this Jesus stuff but can I have some free wine and cookies please".
Full disclosure my day job is to write out of tree drivers for Linux :)
How do the Linux and Windows drivers compare on matters related to CUDA?
The _desktop_ situation is worse, though perfectly functional. But I boot into Windows when I want battery life and quiet fans on my laptop.
Which is really kinda of hilarious considering that so much modern hardware requires proprietary firmware blobs to run.
But it means I can't use Wayland. Wayland isn't critical for me, but since NVidia is refusing to implement GBM and using EGLStream instead, there's nothing I can do about it. It simply isn't worth NVidia's time to make Wayland work, so I'm stuck using X. If the driver were open-source someone would have submitted a GBM patch and i wouldn't be stuck in this predicament.
I can't wait for NVidia to have real competition in the ML space so I can ditch them.
Nvidia is pretty much the only remaining holdout here on the hardware driver front. I don't see why they should get special treatment when the 100%-GPL model works for everyone else.
But it is a problem that you can't reliably have out-of-tree modules.
Also, Linus is wrong: there's no reason that the ZoL project can't keep the ZFS module in working order, with some lag relative to updates to the Linux mainline, so as long as you stay on supported kernels and the ZoL project remains alive, then of course you can use ZFS. And you should use ZFS because it's awesome.
That is the bit I'm trying to get at. Yes it would be best if ZFS was just part of Linux, and maybe some day it can be after Oracle is dead and gone (or under a new leadership and strategy). But it's almost beside the point.
Every other OS supports installing drivers that aren't "part" of the OS. I don't understand why Linux is so hostile to this very real use case. Sure it's not ideal, but the world is full of compromises.
That shouldn't actually matter; it should just depend on the license. But millions in legal fees says otherwise.
As a Linux user and an ex android user, I absolutely disagree and would add that the GPL requirement for drivers is probably the biggest feature Linux has!
Android did start making this less of problem with HAL and stuff, but it's still a problem, just a less big one.
A shim layer is a poor legal bet. It assumes that a judge who might not have much technical knowledge will agree that by putting this little technical trickery between the two incompatible works then somehow that turn it from being a single combined work into two cleanly separated works. It could work, but it could also very easily be seen as meaningless obfuscation.
> Why are all drivers expected to use the GPL
Because a driver is tightly depended on the kernel. It is this relationship that distinguish two works from a single work. A easy way to see this is how a music video work. If a create a file with a video part and a audio part, and distribute it, legally this will be seen as me distributing a single work. I also need to have additional copyright permission in order to create such derivative work, rights that goes beyond just distributing the different parts. If I would argue in court that I just am distributing two different works then the relationship between the video and the music would be put into question.
A userspace software is generally seen as independent work. One reason is that such software can run on multiple platforms, but the primary reason is that people simply don't see them as an extension of the kernel.
The issue here is which parts of the kernel API are allowed for non-GPL modules has been decided to be a moving target from version to version, which might as well be interpreted as "just don't bother anymore".
It's a feature, not a bug. Linux is intentionally hostile to binary-blob drivers. Torvalds described his decision to go with the GPLv2 licence as the best thing I ever did. [0]
This licensing decision sets Linux apart from BSD, and is probably the reason Linux has taken over the world. It's not that Linux is technically superior to FreeBSD or OpenSolaris.
> Expecting all Linux drivers to be GPL-licensed is unrealistic and just leads to crappy user experiences
'Unrealistic'? Again, Linux took over the world!
As for nVidia's proprietary graphics drivers, they're an unusual case. To quote Linus: I personally believe that some modules may be considered to not be derived works simply because they weren't designed for Linux and don't depend on any special Linux behaviour [1]
> Why are all drivers expected to use the GPL?
Because of the 'derived works' concept.
The GPL wasn't intended to overreach to the point that a GPL web server would require that only GPL-compatible web browsers could connect to it, but it was intended to block the creation of a non-free fork of a GPL codebase. There are edge-cases, as there are with everything, such as the nVidia driver situation I mentioned above.
[0] https://en.wikipedia.org/w/index.php?title=History_of_Linux&...
[1] https://en.wikipedia.org/w/index.php?title=Linux_kernel&oldi...
The problem is already addressed: if someone wants to contribute code to the project then it's licensing must be compatible with the prior work contributed to project. That's it.
With that, they do expose a userspace filesystem driver interface, FUSE. There used to be a FUSE ZFS driver, though I believe it's mostly dead now (But I never used it, so I don't know for sure). While it's not the same as an actual kernel FS driver (performance in particular), it effectively allows what you're asking for by exposing an API you can write a filesystem driver against without it being part of the kernel code.
It is the policy of linux development at work. Linux kernel doesn't break userspace, you could safely upgrade kernel, your userspace would work nice. But Linux kernel breaks inner APIs easily, and kernel developers take responsibility for all the code. So if a patch in memory management subsystem broke some drivers, kernel developers would find breakages and fix them.
> We don't consider Windows drivers part of Windows.
Yeah, because Windows kernel less frequently breaks backward compatibility in kernel space, and because hardware vendors are ready to maintain drivers for Windows.
https://www.kernel.org/doc/html/latest/process/license-rules...
It seems that some people are oblivious to the actual problem, which is some people want their code to be mixed into the source code of a software project without having to comply with the rightholder's wishes, as if their will shouldn't be respected.
> We don't consider Windows drivers part of Windows.
I'm not sure you can commit your source code to the windows kernel project.
It's less a think Linux can work on then a think lawmakers/courts would have to make binding decisions on, which would make it clear if this usage is Ok or not. But in practice this can only be decided on a case-by-case basis.
The only way Linux could work on this is by:
1. Adding a exception to there GPL license to exclude kernel modules from GPL constraints (which obviously won't happen for a bunch of reasons).
2. Turn Linux into a micro kernel with user-land drivers and interfaces for that drivers which are not license encumbered (which again won't happen because this would be a completely different system)
3. Oracle re-licensing ZFS under a permissible Open Source license (e.g. dual license it, doesn't need to be GPL, just GPL compatible e.g. Apache v2). Guess, what that won't happen either, or at last I would be very surprised. I mean Oracle is running out of products people _want_ to buy from them and increasingly run into an area where they (ab-)use the license/copyright/patent system to earn their monny and force people to buy there products (or at last somehow pay license fees to them).
Vendors are expected to merge their drivers in mainline because that is the path to getting a well-supported and well-tested driver. Drivers that get merged are expected to use a GPL2-compatible license because that is the license of the Linux kernel. If you're wondering why the kernel community does not care about supporting an API for use in closed-source drivers, it's because it's fundamentally incompatible with the way kernel development actually works, and the resulting experience is even more crappy anyway. Variations of this question get asked so often that there are multiple pages of documentation about it [0] [1].
The tl;dr is that closed-source drivers get pinned to the kernel version they're built for and lag behind. When the vendor decides to stop supporting the hardware, the drivers stop being built for new kernel versions and you can basically never upgrade your kernel after that. In practice it means you are forced to use that vendor's distro if you want things to work properly.
>[...] nVidia is never going to release full-featured GPL'd drivers.
All that says to me is that if you want your hardware to be future-proof, never buy nvidia. All the other Linux vendors have figured out that it's nonsensical to sell someone a piece of hardware that can't be operated without secret bits of code. If you ever wondered why Linus was flipping nvidia the bird in that video that was going around a few years ago... well now you know.
[0]: https://www.kernel.org/doc/html/latest/process/kernel-driver...
[1]: https://www.kernel.org/doc/html/latest/process/stable-api-no...
To answer your excellent question (and ignore the somewhat unfortunate slam on people who seem to differ with your way of thinking), it is an intentional goal of software freedom. The idea of a free software license is to allow people to obtain a license to the software if they agree not to distribute changes to that software in such a way so that downstream users have less options than they would with the original software.
Some people are at odds with the options available with licenses like the GPL. Some think they are too restrictive. Some think they are too permissive. Some think they are just right. With respect to you question, it's neither here nor there if the GPL is hitting a sweet spot or not. What's important is that the original author has decided that it did and has chosen the license. I don't imagine that you intend to argue that a person should not be able to choose the license that is best for them, so I'll just leave it at that.
The root of the question is "What determines a change to the software". Is it if we modify the original code? What if we add code? What if we add a completely new file to the code? What if we add a completely new library and simply link it to the code? What if we interact with a module system at runtime and link to the code that way?
The answers to these questions are not well defined. Some of them have been tested in court, while others have not. There are many opinions on which of these constitutes changing of the original software. These opinions vary wildly, but we won't get a definitive answer until the issues are brought up in court.
Before that time period, as a third party who wishes to interact with the software, you have a few choices. You can simply take your chances and do whatever you want. You might be sued by someone who has standing to sue. You might win the case even if you are sued. It's a risk. In some cases the risk is higher than others (probably roughly ordered in the way I ordered the questions).
Another possibility is that you can follow the intent of the original author. You can ask them, "How do you define changing of the software". You may agree with their ideas or not, but it is a completely valid course of action to choose to follow their intent regardless of your opinion.
Your question is: why are all drivers expected to use the GPL? The answer is because drivers are considered by the author to be an extension of the software and hence to be covered by the same license. You are absolutely free to disagree, but it will not change the original author's opinion. You are also able to decide not to abide by the author's opinion. This may open you up to the risk of being sued. Or it may not.
Now, the question unasked is probably the more interesting question. Why does Linus want the drivers to be considered an extension of the original software? I think the answer is that he sees more advantages in the way people interact in that system than disadvantages. There are certainly disadvantages and things that we currently can't use, but for many people this is not a massive hardship. I think the question you might want to put to him is, what advantages have you realised over the years from maintaining the license boundaries as they are? I don't actually know the answer to this question, but would be very interested to hear Linus's opinion.
> The root of the question is "What determines a change to the software". [...] The answers to these questions are not well defined.
And that's fair, but what confuses me is that I never see this question raised on non-Linux platforms. No one considers Windows drivers a derivative of Windows, or Mac kernel extensions a derivative of Darwin.
Should the currently-in-development Windows ZFS port reach maturity and gain widespread adoption (which feels possible!), do you foresee a possibility of Oracle suing? If not, why is Linux different?
Perhaps they do, but the difference is that their licensing does not regard their status of derivative works as being important. Those platforms have their own restrictions on what drivers they want to allow. In particular, Mac doesn't even allow unsigned drivers anymore, and any signed drivers have to go through a manual approval process. And don't forget iOS, which doesn't even support user-loadable drivers at all.
>Should the currently-in-development Windows ZFS port reach maturity and gain widespread adoption (which feels possible!), do you foresee a possibility of Oracle suing? If not, why is Linux different?
I'm not sure, I haven't used Windows in many years and I don't know their policies. But see what I said earlier: the simple answer is that the license is different from the license of Linux. For more details, the question you should be asking is: Is the CDDL incompatible with Windows licensing?
He is claiming that it comes down to the user's choice, which would be just fine if that were true. The only problem here is that Linux has purposely taken steps to hinder that choice.
especially so given the recent petulant attitude that broke API compatibility in the LTS branch just to spite the ZFS developers: https://news.ycombinator.com/item?id=20186458
compete honestly on technical merit, rather than pulling dirty tricks that you'd expect of Oracle or 1990's MS
That said, after the Oracle Java debacle, I can see why Linus would not be receptive towards merging ZFS into the kernel. I just wish he argued the point on legal issues alone instead of making up stories about non-existent technical flaws in ZFS. The whole thing is basically a work of art. Oracle should consider GPL-ing it and integrating it into Linux directly.
I think Apache (or BSD/MIT) would be far more palatable, as GPL'ing it would cut off the BSDs as well as OpenSolaris, which would certainly be a bummer.
Isn't FreeBSD now using ZFSOnLinux project as well?
In operations, I treat them as semi-black-box (grey-box?) appliance where they only do the file storage function and are moral equivalents of Network Appliance NAS boxes. I don't try to convince the Linux or Windows teams to migrate other workloads to FreeBSD, and mostly don't want them to since that would mean Yet Another app environment to support.
FreeBSD has a really efficient network stack, so I can attach the FreeBSD stores at 10GB (usually, 2-4 bonded 10GB links) and it keeps up fine. I I have colleagues who are doing the same with 40GB links (40-160BG aggregate) to backbone networks, and apparently there are many shops hooking up FreeBSD with 100GB links. Limiting factor seems to be ZFS and the storage subsystems supporting it, not network, which is interesting, but I don't have the ability to benchmark the 40GB and higher stuff.
I want cheap and reliable snapshots, export & import of file systems like ZFS datasets, simple compression, caching facilities(like SLOG and ARC) and decent performance.
I have heard of durability issues with btrfs, and do not want to touch it if it fails with its primary job.
Or Bcachefs is probably the only thing that might get there.
The amount of engineering hours went into ZFS is insane. It is easy to get a project that has 80% similarity on the surface, but then you spend the same amount of time from 0 - 80% on the last 20% and edge cases. ZFS has been battle tested by many. Rsync is on ZFS.
The amount of Petabyte stored in ZFS safely over the years gives peace of mind.
Speaking of Rsync, normally a topic of ZFS on HN will have him resurface. Hasn't seen any reply from him yet.
Does anyone remember the parity patches they rejected in 2014?
> Your work is very very good, it just doesn’t fit our business case.
I haven't followed it much. Does it have anything more than mirroring (that's stable) these days?
You're saying they should stop supporting a project that was considered stable by the time the other started being developed. Why do that? What makes Bcachefs a better choice?
I don't think too many people consider it stable enough for production, either. (Unless you count a very limited subset of its functionality).
I rather run Bcachefs today than Btrfs, by a mile. At least with bcachefs I won't lose my data.
After that, none of the features like compression, snapshots, COW or checksums meant anything to me. I'm much happier with ext4 and xfs on lvm.
I think they just said: "The on-disk data structure is stable" and lots of people misinterpreted that as "the whole thing is stable"
A stable on-disk data structure just means it's been frozen and can't be changed in non-backwards compatible ways. It says nothing about code quality, feature completeness or if the frozen data structure was any good.
Snapshots are a huge feature for sure, but it's not like bcachefs is completely incapable without them.
There was a very recent update he gave in late December (2019) that mentioned he's actively chipping away at the roadblocks for snapshots.
I think everyone is in agreement that ZFS can't be included in the mainline kernel. The question is just if users should be able to install and use it themselves or not.
The follow up actually clears things up pretty well. https://www.realworldtech.com/forum/?threadid=189711&curpost...
For import export, IIRC XFS has support for it and you can dump/import LV snapshots to get atomicity.
For caching there is LVM cache, should be again possible to combine with thinpool & RAID. Or you can use it separately for normal LV.
All this is functionality tested by years of production use.
For compression/deduplication, that is AFAIK work in progress upstream based on the open sourced VDO code.
Never made snapshots with LVM. Always used LVM as a way to carve up logical storage from a pool of physical devices but nothing more. I need to RTFM on how snapshotting would work there - could I restore just a few files from an hour ago while letting everything else be as they are?
With ZFS, I use RAM as read chace(ARC) and an Optane disk as sync write cache(SLOG). I wonder if LVM cache would let me do such a thing. Again, a pointer for more manual reading for me.
Compression is a nice to have for me at this moment. Good to know that it is being worked on at the LVM layer.
For some reading about LVM thin provisioning:
http://man7.org/linux/man-pages/man7/lvmthin.7.html
https://access.redhat.com/documentation/en-us/red_hat_enterp...
There is difference between 'all these tools have been used in production' and 'this is an integrated tool that has been used for 15+ years in the biggest storage installations in the world'.
There's also Lustre but it's a different beast altogether for a different scenario.
Once you actually use them, you discover all the ways that btrfs is a pain and zfs is a (minor) joy:
- snapshot management
- online scrub
- data integrity
- disk management
I lost data from perfectly healthy-appearing btrfs systems twice. I've never lost data on maintained zfs systems, and I now trust a lot more data to zfs than I ever have to btrfs.
Granted, at enterprise scale this hardly matters because you can just send-receive to rebuild pools if you have enough spares, but for consumer-grade deployments it's a non-negligible annoyance.
Twice btrfs ended up in a non-mountable situation, but both times it was due to a known issue and #btrfs on freenode was able to walk me through getting it working again.
With ZFS, I neded up in a non-mountable system, and the response in both #zfs and #zfsonlinux to me posting the error message were, "that sucks, hope you had backups." Since I both had backups and it was my laptop 2000 miles from home that was my only computing device, I didn't dig deeper to see if I could discover the problem. FWIW, I've been using ZFS on that same hardware for almost 2 years since with no issues.
> I lost data from perfectly healthy-appearing btrfs systems twice.
I still consider btrfs as beta-level software. This is why I never looked into it very seriously and asked this question.
Looks like btrfs has something around five years to be considered serious at the scale where ZFS just starting to warm-up.
But in the end it always turns out that only if you 'use' it correctly it is actually not gone eat your data.
I used ZFS for far longer and had far fewer issues.
Once a little more guidance comes out about how to properly use VDO and Stratis together, I'll move my personal stuff to it.
There is also BeeGFS, I haven't used it but /r/datahoarders sometimes touts it.
Not for linux but I have been keeping an eye on M Dillons DragonFly BSD where he has been working on HAMMER2, which is very interesting.
I don't know much but bcachefs has been making more waves lately also.
I think the bottom line is that people need to have good backup in place regardless.
btrfs still has a write hole for RAID5/6 (the kind I primarily use) [0] and has since at least 2012.
For a filesystem to have a bug leading to dataloss unpatched for over 8 years is just plain unacceptable.
I've also had issues even without RAID, particularly after power outages. Not minor issues but "your filesystem is gone now, sorry" issues.
Pretty much all software-raid systems suffer from it unless they explicitly patch over it via journaling. Hardware raid gets away with it if it has battery backups, if they don't they suffer from exactly the same problem.
I thought I wanted RAID5, but after reading horror stories of drives failing when replacing a failed drive, I decided it just wasn't worth the risk.
I currently run RAID1, and when I need more space, I'll double my drives and set up RAID10. I don't need most of the features of ZFS, so BTRFS works for me.
If a disk fails and resilvering causes a cascading failure, I can restore from a backup.
I think you might be mistaking RAID for a backup, which is a mistake. RAID is very much not a backup or any kind of substitute for a backup. A backup ensures durability and integrity of your data by providing an independent fallback should your primary storage fail. RAID ensures availability of your data by keeping your storage online when up to N disks fail.
RAID won't protect you from an accidental "rm -Rf /", ransomware or other malware, bugs in your software or many other common causes of data loss.
I might consider RAID10 if I were running a business-critical server where availability was paramount, or where I needed decent random read/write performance but even so I'd still want a hot-failover and a comprehensively tested backup strategy.
Hardware RAID has very poor longevity. Vendor support and battery backup replacement collide in BIOS and host management badly.
Disclaimer: I work on Dell rackmounts, which means rather than native SAS I am 'Dells hack on SAS' which is a problem and I know its possible to 'downgrade' back to native.
Somewhat recently I dealt with LSI and Dell cards. Longevity seemed just fine for a normal 3 year server lifecycle. The only time we had an issue is when the power went down in the data center. The power spike fried a few of the cards. Luckily we had spares.
Way way back I dealt with the Compaq/hp smartarrays. Those were awful. Also anything consumer grade is awful.
Btw, ZFS scrub is not only a RAID-block-check but also a partial fsck, so its not really comparable.
I have a strong feeling Linus has never actually used ZFS.
Certainly about a decade ago the ZFS' chief architect, Jeff Bonwick, and Linus have met:
Honestly Linus's attitude is refreshing. It's a sign that Linux hasn't yet become some stiff design-by-committee thing. One guy ranting still calls the shots. I love it. Protect this man at all costs.
Pretty sure this is false. ZFS does support trim (FreeBSD had trim support for quite a while, but ZoL has it now as well), as well as supporting l2arc and zil/slog on ssd.
> ZFS partitions are also almost impossible to resize
You can grow zfs partitions just fine (and even online expand). You just can't shrink them.
That's not even entirely true, though it requires shuffling around with multiple vdevs temporarily and doesn't presently support raidz. Also vdev removal is primarily made to support an accidental "oops, I added a disk I shouldn't have" rather than removing a long-lived device -- there's no technical restriction against the later case, though the redirect references could hamper performance.
The official stance has always been to send/receive to significantly change a pool's geometry where it isn't possible online.
I'm not sure you've actually used ZFS very much as any way I can see you could be meaning this, it is actually pretty straightforward and simple to resize partitions with ZFS pools and volumes within ZFS pools.
For example, if you mean that you have a root zpool on a device using only half the device, you just have to resize the partition and then turn on `autoexpand` for the pool.
It's kind of apples to oranges, really.
FreeNAS documentation[0] makes it pretty clear.
In ZFS, you cannot add devices to a vdev after it has been created -- however, you CAN add more vdevs to a pool.
So basically, your complaint is that ZFS wants to have stripes of vdevs and that instead of adding 1 drive to a 3 drive RAID5 to make a 4 drive RAID5, you have to add 3 drives to a RAIDZ1 for a 6 drive RAIDZ+0 that is equivalent to a RAID50 on a hardware controller.
Yes, it's more enterprisey, but it's not especially more difficult and the result is different and perhaps better depending on your use case.
What does that mean?
I'll drop ZFS the moment I have an alternative with the same features:
- disk management with simple commands that can create raids in any modern configuration
- zero cost snapshots
- import/export (zfs send/recv)
- COW and other data integrity niceties
- compression, encryption, dedup, checksums
I am very grateful to the OpenZFS community, and I think they deserve praises for their work. Saying the code is not maintained is quite unfair.
IMO it's best viewed as a legal positioning of professing ignorance such that Oracle's thugs don't go after him for "copying" ZFS features, knowingly developing software that will be mixed with CDDL code, etc.
It's similar to how it's not a good idea for an engineer to read patents.
Has Linus not seen the work that the OpenZFS folks are doing?
ZFS is amazing and I would soon go to a BSD flavor with a fun set of user land utilities than give it up.
This news would come as a surprise to the folks at LLNL who work on ZoL:
That's what he meant by Oracle licensing issues. Java API infringement case against Google
ZFS doesn't primarily target single disks and small arrays anyway. :)
rotational disks are getting cheaper and cheaper, 10TB disks in two years might cost as little as 2TB disks today (i got a 2TB disk for like 50€ off Amazon).
Yes, but without the array as you stated. We have 300+ 10TB disks at our datacenter today and, ZFS is relevant at this disk count, I/O and client load.
Running ZFS at small scale is raising a cow at home for a bucket of raw milk. It's more of a fun curiosity rather than a production level operation.
I'd run LVM or md or something similar at home instead of a full blown ZFS setup for practical reasons.
ZFS was freely relicensed under the CDDL by Sun. Oracle can do nothing to take back any of the rights granted under the terms of the licence retrospectively. They haven't got any grounds whatsoever to curtail anyone's use or modification of the ZFS code.
But it is a separate point he made about using zfs in general, and it's certainly not correct if you take one look at the activity in the zfsonlinux project on GitHub.
There is no way, after any technical evaluation by himself he would come to those conclusions.
No-one's seriously concerned that adopting OpenJDK could land them in legal trouble, as far as I know. I don't think a separate legal entity is always necessary.
Source: https://www.techrepublic.com/google-amp/article/microsoft-ma... https://techcrunch.com/2019/01/17/google-remains-the-top-ope...
So, I use Linux because Docker and BTRFS work just fine for me use case. I prefer FreeBSD, but unfortunately I'm unable to solve my problems easily with just FreeBSD, so I'm using something else.
That's why we have different software.
If you asked Theo to merge an encryption algorithm for example into OpenSSH and OpenBSD - he's going to have an opinion about it - and that's his thing.
Why would this be controversial at all?
Linus can do what he wants in regards to his branch (which because he's the lead, becomes official Linux), but there's no reason any one else (or any distro) can't do the integration. That's how open source works!
Of course, whoever does the integration may incur Oracle's wrath. Tread at your own discretion. If those people like it so much that they will put up their own money when Oracle's lawyers come calling, that's completely up to them.
In my opinion, people who constantly clamour for such things against the technical judgment of open source maintainers are freeloaders. They can propose ideas, but just because the maintainer doesn't want to do it doesn't mean they can scream bloody murder. Just put up your own money and fork it and/or maintain your own fork, which is exactly what the ZFS on linux community is doing - which is the right thing.
Anyone else can work with the ZFS on linux maintainers to take a bit of the burden on, whether it's rebasing or updating docs on how the integration works, etc. It's a group effort.
Anyone using Stratis? Just noticed it recently went to 2.0. https://en.wikipedia.org/wiki/Stratis_(configuration_daemon) Curious to know how it handles device failures, removals, additions and so on.
I'm not involved with the project in any way, apart from sending him a few bucks a month on patreon. It's literally the only open source thing I sponsor; it seems like a really worthwhile effort especially considering Linus' advice here...
ZFS is certainly "nice to have" on desktop, but the main use case is going to be servers and NAS. You can use BSD there, it won't bite.
Also, while ZFS for me has been performant, that seems like a silly reason to decide to use it or not use it. I think ZFS pools and snapshots would be among the deciding factors to use it or not.
FWIW, as some other commenters have said, I'd rather drop Linux than drop ZFS. I'm actually only even running Linux on my home server right now because I decided to try Proxmox out on it months ago and it was soooo obscenely easy to install I haven't bothered to reset it yet (though I need to for various reasons; Proxmox itself being the first, ha).
Really all I care about for my host OS these days is the ability to do virtualization and GPU pass-through... Linux is an option but not the only one. Having a robust storage system where drives can fail, exist as one logical drive, are replicated, has snapshots (including RAM), and I literally don't have to worry about it-- that though, that's really only available with ZFS.
Either way, BTRFS works for everything I need it to do and it's native.
"Don't use ZFS. It's that simple. It was always more of a buzzword than anything else, I feel, and the licensing issues just make it a non-starter for me."
Yeah, just use XFS or EXT4 without ANY data consistency ... you can use FAT32 also, similar level filesystem.
Linus seems out of touch on this one IMHO.
My use-case is that I run Sandstorm, and want to be able to back it up while it's running. That means:
- Ensure there aren't any existing snapshots
- Take a snapshot
- Mount the snapshot as a filesystem
- Run tarsnap against that filesystem
- Release the snapshot
I think we don't see this normally happening; situations in which the (technical) responsible for the Apple or MS kernels get to answer question and explain with such level of detail.
I think also that is even more interesting than the original blog post that originated that thread. Someone should harvest all these Linus comments and order them in some kind of "lectures" library.
If you use a filesystem that isn't mainline and it breaks it's all on you to figure it out and fix it. Having used experimental filesystems before and been burned I would rather stick with what I know and won't change overnight.
I'll keep an eye open for new filesystems of course but if it's not mainline unless it's for personal hacking no.
> I own several SBC's (Small Board Computer) that do not run mainline kernels. The company that makes them provides their own patched kernel so if it breaks I'm up creek but I know who to bitch at.
Money doesn't solve all problems, but money can solve legal problems c/f the lawsuits people like newegg do, to get rid of the IPR leeches. (And cloudflare?)
I've also heard success stories using ZFS on Linux, but I haven't bothered because I'd just rather use something that's in the kernel instead of something outside it.
I would think the wide availability and usage of ZFS and the fact that this is no longer 20 years ago when companies would sue as if they were 19th century tycoons there must be some sort of "well you let ZFS be used this long without suing you can't just let it be open for so long and then sue when it is profitable" sort of statue in American law. Again I'm not a lawyer and I know enough about law to know I do not know enough about law.
Here, Linus is putting emphasis on license, not technical whatever details of ZFS. He clearly doesn't use ZFS and is not even interested in the problem ZFS solves. He is only "interested" is his (and community's) control over the ZFS source code.
So, his logic basically becomes this:
ZFS is not mainline-able, so veto it until Oracle changes its attitude - a simple old FOSS infestation tactic.
So, please, move on, people. The discussion is not even about file system...
This seems pretty ironic, given that the whole problem is Linux developers trying to claim and enforce that only other GPL code is allowed to use its APIs (which, IMHO, goes beyond both the intent and letter of the license). The issues aren't exactly the same, but Linux sure seems to be a lot closer to Oracle than to Google here.
Can never trust someone who easily changes their licenses from private to public. Who knows in future they might change it back to private!?
furthermore, the allure of ZFS means people aren't testing their disaster plans until it's too late, bc ZFS is "resilient".
lastly, data recovery is expensive as all hell if even possible. i am talking order of magnitude four figures for 100s of GBs and sketchy probabilities.
ZFS is the ultimate "pet" in the pets vs. cattle continuum. in a world where shoddy engineering and "break things fast" is the zeitgeist, i'm happy to use a classic dumb FS like ext4 and pathologically backing it up and testing said backups.
i would not risk any of my personal treasured data to ZFS due to inherent existential threats. i would implore ZFS users to evaluate and test their setups, and especially use ECC RAM - like, starting now - to protect their assets.
This is FUD. ZFS does as well, if not better than the average file system with its focus on integrity, online scrubs etc. On the other hand "use ECC RAM" is standard best practices for any mission critical data, no file system magic is going to fix computer RAM lying to you 100% of the time. Its the standard recommendation for ZFS because its rare to be deployed in environments that can tolerate data corruption.
> pathologically backing it up and testing said backups.
ZFS doesn't remove the need for backups and no one seriously makes that arguments. Though snapshots + send/receive make them very easy to do in ZFS.
Live storage is never 'cattle', that is idiotic your running filesystem IS actually a pet. Harddrives are 'cattle' and that's exactly what ZFS treats like 'cattle'.
ZFS was born out of long frustrations with file system and was systematically designed to protect against data corruption and bad hardware. It is literally the exact opposite of 'move fast and break things'.
Go and actually watch the videos where the designer show it for the first time. It speaks very clearly about how and why they designed it.
> i would not risk any of my personal treasured data to ZFS due to inherent existential threats. i would implore ZFS users to evaluate and test their setups, and especially use ECC RAM - like, starting now - to protect their assets.
ZFS has always recommended ECC to its users. No filesystem can protect you from not having it.
https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...
With e.g. ext4 you can get back to a mountable state pretty much guaranteed with e2fsck. You might loose a few files, or find them in lost+found, etc. but at least you have something.
The reason ZFS doesn't have a offline repair tool is pretty convincing. Once you have zettabytes (that's the marketing) of data, running that repair tool will take too long, so you'd have to do everything to prevent that in the first place anyway. Including checksumming everything, storing everything redundantly and using ECC RAM.
ZFS does indeed catch memory errors. If you are running without ECC, most filesystems will happily write that corrupt data to disk. Unless the corruption is in the metadata, you will be none the wiser.
It's not a backup by itself, but it makes a fine backup target if it's located somewhere else, since it's both redundant (hard to lose data by accident) and snapshotted (hard to lose data by mistake) - it was my local CrashPlan target (alongside cloud) back when CrashPlan supported home users.
He never agreed to roll over and agree with every technological persuasion, he agreed to be nicer to people. This was nice to people, but rude to a technology (ZFS), that seems consistent.
Most of his worst rants were directed at maintainers who commited changes that resulted in bugs in the kernel.
The OP of this thread is from a user, so perhaps we need to wait until another big bug gets commited to see the results of his hiatus.