OpenZFS vs. Btrfs and other file systems (2017)
ixsystems.com
ixsystems.com
> Time Machine can’t back up to an APFS-formatted disk. If you select a back up disk formatted as APFS, Time Machine offers to reformat it as HFS+.[1]
[1] https://support.apple.com/guide/mac-help/types-of-disks-you-...
It’s a big IF to depend solely on flash based error checking. I think Apple likely figured the cost of implementation and system overhead wasn’t worth it for Macs and iOS devices.
I’d be very surprised if they use APFS or a non checksumning file system on their iCloud server farm though.
Part of the problem is that the app might know what the proper checksum is and it would be nice to be able to write a document ahead of time and tell the FS that you already know the checksum.
Won't save a massive amount of CPU but it would be worth it if it was easy to implement.
Whether or not you have a RAID you should still have backups, and you don't want to replace a good backup by a corrupt one.
I had a VM running OpenSuse Tumbleweed - I know, rolling release, don't expect stability, etc - and it would consistently freeze after being up and unused for a few days. Rebuilt the VM but used XFS instead of Btrfs, it was rock solid. This happened within the last four months.
At work, we have a number of machines using Btrfs. At some point they get into a state where the weekly Btrfs scrub process hangs the machine for hours at a time. The solution in every case was to move that filesystem onto a non-Btrfs volume. This is an ongoing issue, but I'm not in the team responsible so I don't know what they're doing with this. Anecdotally, this seems to be a problem when there is a lot of writing going on with the volumes. Machines that use Btrfs root but do most of their work on NFS volumes don't seem to experience this issue.
I also have a laptop running OpenSuse Tumbleweed with Btrfs. One day it somehow got itself into a state where it would hang when mounting the filesystem read/write, but it would be fine if it was mounted read-only. I can't recall exactly what happened to cause it, but I'm pretty sure I didn't do anything exceptional with the computer - I think I rebooted it from the GUI while it was doing a scrub in the background, but that's just a guess. I think this happened about a year ago.
This is all disappointing because some of the things Btrfs enables are awesome. In particular, OpenSuse's snapper has saved me countless hour of work. I haven't played with the send/receive stuff, but it seems like a great solution for backups.
In the meantime, I'm using ZFS on a variety of FreeBSD boxes with a variety of workloads (including one running more VMs than its RAM should support) and it's been rock solid.
At this point it will be a long time before I give Btrfs another look. It has the potential to be really good, and I want it to be good, but I don't believe it's there yet.
A great example is that if you ever try to run a database or a large virtual machine on a btrfs mount, you'll quickly become acquainted with chattr +C for "nocow". Any write-intensive applications grinds the filesystem to a halt until you disable the copy-on-write functionality for that file specifically, making it so snapshots, etc., are ineffective and the FS stops tracking the changes that would make them possible.
Whereas on ZFS, like any application that uses fixed block sizes, virtualization and databases _may_ require some alignment tuning to avoid a read-modify-write cycle, and ZFS works without fuss from that moment on.
I know that Facebook is investing heavily in btrfs right now, but to be frank, I don't see the point. btrfs doesn't have its youth as an excuse anymore and the core issues that exist seem like they're pretty fundamental.
It's a gargantuan waste of effort when people could just take the tack that Ubuntu has taken and decide all that FUD around the CDDL, kicked up by fears that the commercial Unices were plotting to take their revenge on Linux, that maybe all that isn't super relevant anymore, and they can just start shipping ZoL.
Today I'm running ext4 because maintaining ZFS setups is exhausting since it is not upstreamed.
Still... I hope there's a future in one of these volume-manager-meets-filesystem filesystems. Maybe bcachefs will end up being the answer some day?
Although looks like it'll be DragonFly only.
Also the author do not mention that LLNL zfs Linux port is not as production ready than FreeBSD one.
Personally I use nilfs2 on top of lvm since is Linux mainline from many many years, it a logfs like Hammer volumes, so it protect effectively against accidental overwrite/deletion and can be live resized both in grow and shrink so coupled with LVM can be a poor man's pooled storage... But Hammer on Linux will be a success like OpenSSH!
Got it migrated over to HAMMER 2 recently and it's pretty sweet.
Here's some benchmark comparison - https://www.phoronix.com/scan.php?page=article&item=8-linux-...
Windows is even worse with NTFS, which is a complete pile of garbage. I've fixed hundreds of computers with NTFS corruption over my lifetime and it's mind-boggling that Microsoft is fine with this shit.
It's fairly crash resistant, has well worn and battle-tested codepaths and generally seems to not loose data outside bitrot. So why pick btrfs with it's dataloss stories or ZFS with it's complex management?
It's an effort to switch to a new FS as default, especially if either itself or the integration is not as well tested or completed yet.
So everyone just sticks to the thing they know how it sucks.
Compared to just running gdisk and using raw partitions - sure, but I’ve been using LVM for over a decade now.
Even Synology and QNAP now have plug-and-play solutions to expand a RAID one disk at a time. You can even start plugging in higher capacity disks and once you have enough extra parity, the higher capacity becomes used magically.
No need for wasteful RAID10/50/60 setups. Just plug in more drives.
This also affects the RAID1 setup, any storage expansion requires 2 disk to pull up a new RAID1 into the pool.
I don't know where you read that, but don't be fooled. That functionality has been "in the works" for quite a long time. But it hasn't yet appeared.
In the past it has been said that something called "block pointer rewrite" was a prerequisite. Just google for that.
Here's some 10 year old information about it. At that time it was already a "frustratingly frequent request." I stopped searching when I found something 10 years old, but I'm sure you with some effort you could go back much further than that.
Those RAID set ups are only "wasteful" if you don't need that level of redundancy (ie the number of disks that can fail at the same time). Most home users only really need 1 redundancy disk (actually most home users don't even need that) but a production database server is another matter entirely.
For clarity, the reason we have redundancy drives isn't to mitigate the need for backups; it's the mitigate the likelihood of downtime while you rebuild your storage array from a backup. Which is why most home users don't even need 1 redundancy disk - they just need to be disciplined in backing important stuff up.
Scenario 1: Replace smaller disc with bigger disc. Add new disc as mirror to vdev. Wait for it to resilver. Remove original disc from vdev. Pool will autoexpand to new disc size or can be manually expanded otherwise.
Scenario 2: Expand virtual storage. Use LVM lvresize, or expand VM virtual disc. Pool will autoexpand to new volume size; may need manual triggering if the system doesn't pick up the new device size automatically. Or you can do the same as for physical discs. Add new virtual storage as mirror, resilver, then drop the old storage.
You don't need a handful of tools just to do one simple task, just hack away at the zfs and zpool commands and everything gets done. No need for parted, mdadm, lvm or support of your needs compiled into the kernel. No matter on which operating system you are currently, you only need those two tools and they work the same everywhere.
Expanding a RAID vdev is not possible, instead the recommendation is to waste 50% of storage for a pool of mirrored devices to get the same protection as a RAID1 of two drives, which isn't much for some people.
For the avarage home user, such a scaling option is already no longer viable.
This is topped with ZFS' poor performance under low RAM conditions. FreeNAS barely managed 10MB/s when running with 2GB of RAM and an old Pentium 4 Dual Core. Not everyone can afford to throw hardware at the problem.
On the other hand, Linux performs under the same conditions with 8 times the speed for simply copying files and mdadm/lvm/unraid allow me to dynamically grow the pool of harddrive space a single drive at a time.
(plus mdadm/lvm is included in basically every larger distro by now, no need to install it or compile support into the kernel)
Or make a new vdev. It takes a bit of planning, but it's not some massive hurdle.
>This is topped with ZFS' poor performance under low RAM conditions. FreeNAS barely managed 10MB/s when running with 2GB of RAM and an old Pentium 4 Dual Core. Not everyone can afford to throw hardware at the problem.
FreeNAS is enterprise software, the default settings expect you to read and follow the system requirements. The developers aren't spending their time supporting it on less powerful systems, but ZFS (and likely FreeNAS) can run fine if you configure it for that hardware.
>Or make a new vdev. It takes a bit of planning, but it's not some massive hurdle.
I had a 3 disk, 6TB RAIDZ (3x 3TB drives). I bought a fourth 3TB disk when I started running low on space, not knowing that this limitation existed. I ended up using an intermediate, 8TB external drive to copy data off, re-create the RAIDZ with 4 disks, and then copy the data back.
This sucked. What should I have done instead?
RAIDZ expansion is coming though[0]. This is the right answer to your problem, but an answer at some point in the future doesn't help you with your problem today.
And, replacing disks one by one would require me to buy 3 disks instead of 1. I did that the last time I grew my array :-/
You should have read almost anything about ZFS before investing several hundred dollars and your data into it. It sucked for you, but seems like a reasonable ask.
I'm surprised you chose ZFS without knowing about it, it seems to be mentioned in every discussion about ZFS.
I was also making the point that "make a new vdev" doesn't solve that particular problem.
Overall, I'm quite happy with ZFS.
The hardest part on Linux if you're running a distro that doesn't officially support ZoL so you'd have to compile you're own kernel module. But even that isn't the problem it used be in the old days of Linux since we usually have bootstrapping scripts to handle the pain now (however if you can run FreeBSD then that's always a good choice as well. It's genuinely a really nice platform to use).
ZFS isn't quite easier to user for multiple disks or volumes, rather, there is a very tiny set of configurations that everyone recommends (RAID1 mirrors, leaving consumers unable to grow pools 1 disk at a time because ZFS is fundamentally unable to grow a RAID vdev). LVM and mdadm manage these jobs much better and are composable to your needs.
I've read people say that but I've never had any issues when I used to run ZFS on various systems with 2GB RAM. So I wonder how many of those people actually had practical experiences running ZFS on low memory systems and how much of it is people just looking at some of the recommended memory specifications (which are high) and start making assumptions about it's performance on low memory systems.
> RAID1 mirrors, leaving consumers unable to grow pools 1 disk at a time because ZFS is fundamentally unable to grow a RAID vdev
You actually can grow a ZFS vdev - exactly by replacing one disk at a time (albeit you don't get the gained capacity until all the disks in the vdev have been replaced).
What you cannot do is shrink it - but that's not a unique problem to ZFS and is a pretty uncommon use-case anyway. Also you cannot grow a ZFS "raidz" vdev by adding disks - but you cannot do that with any typical RAID5/6 array either. However if you want to add disks to a raidz then you can workaround the RAID5/6 striped parity problem by adding the additional disks into a new vdev and adding that new vdev into the existing ZFS volume (named "tank").
The kind of storage arrays that grow by adding solitary disks work quite different to your typical RAID. They'll often have designated parity drives instead. See solutions like "unraid".
> LVM and mdadm manage these jobs much better and are composable to your needs.
LVM and mdadm are not file systems though, so they only solve part of the problem. The fact ZFS handles volume management, software RAIDing, file system operations (and a bunch of other stuff that your typical fs doesn't yet support (eg snapshotting)) and does so from surprisingly intuitive command line tools is what edges ZFS for me in terms of usability.
Ultimately though, "easier" is such a broad term that we're essentially just arguing personal opinion. :)
I've ran it for real. Once a few more background services start running, ZFS bogs down and keels over. I've had to reboot the NAS dozens of times because the entire filesystem had become totally unresponsive.
>You actually can grow a ZFS vdev - exactly by replacing one disk at a time (albeit you don't get the gained capacity until all the disks in the vdev have been replaced).
Which is not an accaptable answer to the situation I presented, you still have to buy multiple disks to grow the pool, you can't do it one at a time.
>The kind of storage arrays that grow by adding solitary disks work quite different to your typical RAID. They'll often have designated parity drives instead. See solutions like "unraid".
Not only unraid, there are other systems that provide almost this. Syno and QNAP can both grow an array in storage capacity if you add a single new disk, even if it's larger and once you have added enough to provide parity, the additional capacity will become used too.
>surprisingly intuitive command line tools is what edges ZFS for me in terms of usability.
I found the commands anything but intuitive though I guess that's a difference of a few years of experience, I'd agree that would be personal opinion if it's easier.
Was that an off the shelf NAS? If so then I'm guessing it wasn't exactly high spec'ed even putting aside the RAM? eg did it have a 32bit CPU? I've heard that does a massive amount to dent ZFS's performance (I wouldn't know because I've only ever ran on it 64 bit systems).
As I said, I've not had any issues with raidz2 on 2GB RAM. I've even ran virtual machines on one OpenSolaris box with ZFS and 2GB RAM so these weren't always dedicated storage servers either.
edit: just read elsewhere you were running FreeNAS on a Pentium 4. Yeah that would do it. FreeNAS wasn't optimised for low end systems (in fact they outright tell you not to!) and the P4 is pretty low end CPU as well. I still think ZFS would have ran on that but you'd have been better off going with vanilla FreeBSD instead.
> Which is not an accaptable answer to the situation I presented, you still have to buy multiple disks to grow the pool, you can't do it one at a time.
...but you can't do that with most traditional RAIDs either and the few you can do it on require you to take the whole storage pool offline while you grow the pool then resize the file system. That process can literally take several days on some storage servers. So ZFS might require you to add vdevs to the pool (tank) but at least it can all be done on live systems without any downtime. That in itself is a massive boon for ease of use in my opinion.
To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. At least outside of home users anyway. And if you are the type of person that would need to do that then you wouldn't go for a traditional raid5/6 style pool either. Thus the argument becomes a comparison of apples and oranges as you'e comparing completely different types of software RAIDing.
> Not only unraid
I didn't say "exclusively unraid". I said "solutions like unraid". I'm fully aware there are other solutions out there.
> there are other systems that provide almost this. Syno and QNAP can both grow an array in storage capacity if you add a single new disk, even if it's larger and once you have added enough to provide parity, the additional capacity will become used too.
When you said "there are other systems that provide almost this" and "once you have added enough [disks] to provide parity, the additional capacity will become used too." I really had to question what your actual point of this discussion was because just a paragraph earlier you were calling my points "unacceptable" because ZFS required you to add more than 1 extra disk to provide parity; and now you're saying your preferred solutions also does the same?
I hope I'm just being presumptuous here but it's starting to feel like you're just arguing for the sake of arguing. :-/
> I found the commands anything but intuitive though I guess that's a difference of a few years of experience
Maybe. I've been a Linux and UNIX sysadmin for a fair few years now so experience is not something I'm short of. :)
Honest question: what is your experience of ZFS outside of the troubles you had on your NAS? I could totally see why the problems you described would taint your opinion of ZFS but I definitely wouldn't say what you experienced is typical for ZFS.
In any case, I'm not going to argue that everyone should be running ZFS - it's just technology, you can run whatever you want to run.
OpenZFS is currently working on adding code to allow for vdev expansion[1]. I am a fan of ZFS as well, but I don't get not wanting to admit its limitations. Not being able to expand vdevs is a limitation and it's being addressed.
Or in other words: if growing the vdev is a hard requirement then a traditional raid might not be the best option regardless of ZFS. It’s about picking the right tool for the job rather than moaning that ZFS doesn’t do every job perfectly (which seems to be the argument the GP was making).
No it was a repurposed office computer running a 12 year old Pentium 4 Dual Core (about 2.8GHz per Core). 2GB of DDR2.
As you mention, FreeNAS isn't optimized for that but a plain BSD install with ZFS didn't fare much better either and required a rather inane amount of work compared to other solutions that worked OOTB.
>...but you can't do that with most traditional RAIDs either and the few you can do it on require you to take the whole storage pool offline while you grow the pool then resize the file system.
If you use mdadm and ext4 you can do it online.
>To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. At least outside of home users anyway.
There is a lot of home users. People will use RAID5s and 6s at home, they'd rather have no-though-just-works solutions.
The home desktop is possibly the largest filesystem market next to "disposable virtual machine".
>and now you're saying your preferred solutions also does the same?
They don't behave the same. In these solutions if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage.
On ZFS with a RAIDZ1 you'd get 5TB of storage since you'd have to stand up two RAIDZ1 vdevs.
>what is your experience of ZFS outside of the troubles you had on your NAS?
I've been running a few hypervisors using ZFS as storage backend, though at the moment I'm migrating to a LVM thinpool based backend. I've had a few share of problems on that as well, notably ZFS at some point consume almost all of the 32GB of memory and running the OOM killer in the linux kernel until everything had crashed and the host required a hard reboot. Though this hasn't happened since.
That's pure conjecture. Plus I actually have ran ZFS on a Pentium 4 on FreeBSD. For a while that was my personal backup server.
> If you use mdadm and ext4 you can do it online.
Not according to any of the docs I've read in the past. Happy to be proven wrong if you have a citation though.
> There is a lot of home users. People will use RAID5s and 6s at home, they'd rather have no-though-just-works solutions.
> The home desktop is possibly the largest filesystem market next to "disposable virtual machine".
I don't disagree with that but how many of those users are running LVM + mdadm? And how many of those people are just using unraid (or similar purpose built solution for home users)? How many users are just using an off-the-shelf NAS like QNAP et al?
The issue I have with your complaints is that you're flip-flopping across a multitude of industries from desktop to enterprise, which seems grossly unfair for making a balanced opinion. I'm not going to deny that ZFS will lack some desirable features compared to some speciality tools but like I said before, it's about picking the right tool for the job rather than arguing that ZFS is garbage because it doesn't win every single argument in every specific speciality.
I mean you're not even willing to take on board any different configurations of ZFS nor alternative platforms it ships on - instead only basing your biases on one failed attempt at running FreeNAS (something which openly states they need beefy hardware to run) on equipment that is several generations old while then comparing that one specific ZFS instance against every alternative tool you've read about. That hardly seems fair, don't you think?
> They don't behave the same. In these solutions if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage.
No raid would give you 3TB of extra storage when you add a 2TB HDD. That would be breaking the laws of physics because you're now asking your server to create 1TB from thin air.
But to assume you meant 2TB of extra storage when you add a 2TB HDD: in your situation you don't get redundancy so your data becomes at risk if you have a failure. You can actually expand your ZFS tank the same way if you really wanted (but no-one who cares about their data would want to) by adding a HDD vdev (unraided) to an existing tank.
The issue is when you want to expand an existing raidz while retaining redundancy. To do that on any of the existing FOSS tools I've used, you'd have to either buy more than one HDD or do an offline restripe (as is the case with mdadm). I guess at least mdadm allows you to do it - even if it takes the storage pool offline for several days. Personally I've always opted to buy multiple drives and expand the pool (tank) by adding an additional raidz vdev because that not only expands the storage array but also then gives me another failure option (eg I can then lose 2 disks simultaneously without losing data). As you start to expand the number of disks you have, you equally increase the likelihood that more than 1 disk might fail. So having that extra redundancy disk is desirable to me.
> On ZFS with a RAIDZ1 you'd get 5TB of storage since you'd have to stand up two RAIDZ1 vdevs.
Actually no. You have to stand up two vdevs but a vdev can be a raidz, mirror, a solitary HDD or even a file on an existing file system. So you can just add an additional HDD to a ZFS tank but you wouldn't get redundancy on that additional HDD (which is why nobody suggests you should do it).
> I've been running a few hypervisors using ZFS as storage backend, though at the moment I'm migrating to a LVM thinpool based backend. I've had a few share of problems on that as well, notably ZFS at some point consume almost all of the 32GB of memory and running the OOM killer in the linux kernel until everything had crashed and the host required a hard reboot. Though this hasn't happened since.
Yeah, I don't much like how ZFS's cache isn't shared with the kernel as freeable RAM. I get why they did it that way and generally it's not an issue (aside monitoring / reporting tools) because the ZFS kernel module should deallocate memory itself when it detects system memory is getting low but clearly that didn't happen in your case (which is strange - sounds like you found a bug that was worth reporting). Frankly speaking, I'm not a massive fan of ZoL anyway. I've got it working before but honestly I much prefer running ZFS on FreeBSD or Solaris / OpenSolaris,
With regards to your particular use case: I wouldn't have recommended running ZFS for your hypervisor backend anyway. You don't really want CoW for write-heavy processes - particularly ones that have their own journals and/or snapshotting systems such as databases and hypervisors would. Like I said before, it's about using the right tool for the job and XFS would be my preferred Linux file system in those particular instances.
https://raid.wiki.kernel.org/index.php/Growing
ext4 online resize works by invoking resize2fs.
>I don't disagree with that but how many of those users are running LVM + mdadm? And how many of those people are just using unraid (or similar purpose built solution for home users)? How many users are just using an off-the-shelf NAS like QNAP et al?
QNAP and Synology don't do magic. They deploy mdadm, LVM and ext4 or btrfs (though btrfs only in single drive mode, they use LVM and mdadm to span the volume for btrfs).
>That hardly seems fair, don't you think?
It's the real world experience I had with the software and I think it's totally fair to point out and complain about the shortcomings.
>No raid would give you 3TB of extra storage when you add a 2TB HDD. That would be breaking the laws of physics because you're now asking your server to create 1TB from thin air.
Reread what I wrote, I mentioned adding 2x2TB not 1x2TB.
> in your situation you don't get redundancy so your data becomes at risk if you have a failure
Still having redundancy is exactly the point. The resulting array in the example would span 6x1TB and 2x1TB in RAID5, giving me a single drive for parity in each span and resulting in an effective RAID5 for the entire RAID. I can loose any drive in that 4x1+2x2TB array without loosing data.
And it does that online without having to reboot or even halt any ongoing file operations. The volume will simply increase in size once it has finished formatting the new drive and integrated it into the array.
>So you can just add an additional HDD to a ZFS tank but you wouldn't get redundancy on that additional HDD (which is why nobody suggests you should do it).
Again, what Syno offers (and what can be replicated on any Linux system) allows you to add a single drive to expand the storage without loosing parity and more drives to span any drive increases with parity. This allows adding a single drive at a time and at worst you loose any additional capacity without parity yet (in the 1TB/2TB example, that would be loosing 1TB of storage until the second drive is added)
That's resizing the filesystem not the pool (eg `mdadm --grow`). You're also missing the point about what I said. To run those commands you must first `umount` to pool. Hence it being an offline job.
> QNAP and Synology don't do magic. They deploy mdadm, LVM and ext4 or btrfs (though btrfs only in single drive mode, they use LVM and mdadm to span the volume for btrfs).
That's completely irrelevant to the point I was making. How many home users do you know who configure their own home raids using LVM and mdadm?
> Reread what I wrote, I mentioned adding 2x2TB not 1x2TB.
You actually said neither. You said, and I quote, "2TB" (ie without any accumulation prefix).
> Still having redundancy is exactly the point. The resulting array in the example would span 6x1TB and 2x1TB in RAID5, giving me a single drive for parity in each span and resulting in an effective RAID5 for the entire RAID. I can loose any drive in that 4x1+2x2TB array without loosing data
You can actually do that in ZFS. In fact what you've just described were is the exact solution that you called "unacceptable" in ZFS!
> And it does that online without having to reboot or even halt any ongoing file operations. The volume will simply increase in size once it has finished formatting the new drive and integrated it into the array.
Indeed. Lots of commercial tools do support that. I thought we were comparing ext4, LVM and mdadm to ZFS though?
I'm guessing you just glossed other my post where I said you were flip-flopping all over the place? It's easy to pick any specific tool ("tool a")and compare it to a thousand other tools and argue that "tool a" is crap because it doesn't work as well as tool b, c and d in very specific edge cases but all you're doing is trying to win an argument via death from a thousand papercuts rather than forming a sane basis for analysis.
To be honest I think it's probably best we give up on this discourse because your points are all over the place. This feels very much like you've got an axe to grind and no practical experience to back up your prejudices.
No you don't need to unmount the pool under mdadm. You can do a live resize of the array. See the link I posted.
>How many home users do you know who configure their own home raids using LVM and mdadm?
Not many, hence me mentioning Syno and QNAP.
>You said, and I quote, "2TB" (ie without any accumulation prefix).
I said, quote, "if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage."
>In fact what you've just described were is the exact solution that you called "unacceptable" in ZFS!
Again, this is not what ZFS does, the mdadm solution stripes multiple RAID arrays over the same drive, which to my recollection is not a recommended configuration in ZFS by a long shot.
>Lots of commercial tools do support that. I thought we were comparing ext4, LVM and mdadm to ZFS though?
The commercial tools I mentioned previously use ext4, LVM and mdadm native functionality.
By default, sure. There is a FreeBSD (I think) guide to tuning ZFS that lets you tune it for low memory. It's a long time since I've done this so I can't remember which settings, but I ran an old machine tuned for its low max memory, with 4 drives in RaidZ, and it was just fine.
The FreeNAS forum doesn't help much with this as the default opinion there seems to be 16GB ECC. :)
As you mentioned, the bitrot. Also, my understanding is that a COW filesystem can be more reliable than a traditional journaling filesystem in a system crash since nothing is overwritten. Plus there is compression and all of the features provided by a COW filesystem.
The risk of bitrot becomes interesting when you talk about long timeframes like a decade.
Considering the capacity and price of a 2TB disk, compression isn't the immediate need of the consumer either, unless they have a NAS, where Syno and QNAP already provide integrated solutions with btrfs and ext4 both.
The modern ext4 journaling is also very battle tested, it's a very simple and resilient physical journal, data loss in ext4 happens largely when the disk is already failing or has been cheap enough to lie about the state of data being written or not.
For the average consumer that sums up to "it doesn't matter enough". Something like bcachefs could change the game by being btrfs+ZFS but without the dataloss risk.
Of course I’ve been using btrfs RAID1, so perhaps bit rot has been silently corrected.
> The risk of bitrot becomes interesting when you talk about long timeframes like a decade.
I think you are making a lot of assumptions here about frequency of bitrot, especially in the face of failing hardware. Also, consumer desktops are not the only use. There are bunch usages where the risk of bitrot is much more concerning. Or were you intending your original post to be only about consumer usage?
>Considering the capacity and price of a 2TB disk, compression isn't the immediate need of the consumer either, unless they have a NAS, where Syno and QNAP already provide integrated solutions with btrfs and ext4 both.
This kind of feels like you are saying that it isn't needed because it is already being used when it is needed. Am I misunderstanding? Consumers may want compression for smaller hard drives they already own in order to get more out of them. Also, compression can speed up disk IO in some workloads.
> The modern ext4 journaling is also very battle tested, it's a very simple and resilient physical journal, data loss in ext4 happens largely when the disk is already failing or has been cheap enough to lie about the state of data being written or not.
The issue with ext4 resilience isn't about quality of the implementation. It is a fundamental limitation of no COW filesystems. Since things are overwritten, if the system crashes mid write you can end up with partial data. COW filesystems always give you the old data or the new data. This can be worked around but that requires that all of your applications do this workaround.
> For the average consumer that sums up to "it doesn't matter enough". Something like bcachefs could change the game by being btrfs+ZFS but without the dataloss risk.
What makes you think bcachefs is going to be any better? A lot of functionality still has to be implemented. What makes you think that the implementation of this functionality will be any less error prone?
You also haven't addressed the large pile of new features provided by BTRFS/ZFS. Snapshotting alone can make it worth the price of admission for consumers. With snapshotting, you can set up a system where rollbacks after any upgrade can occur seamlessly.
You do know how journalling works, yes? Unless the drive lies about the state of data being written, in which case not even COW will save you, a journal is just as good as COW. Exceptions being coding errors but those can occur on COW with the same devastating result.
Maybe you don't know as much about filesystems as you want to appear to?
Yes, I'm aware of how journaling works. The problem with your description is that it requires both data and metadata journaling. This is rarely the case because data journaling requires that you write all of your data to the disk twice (which also requires additional seeks as well). So, nearly all in-place filesystems default to only metadata journaling for performance reasons (this is the case for ext4). This means that, in practice, if you get a system crash mid file write, you will end up with a partially updated file. On a COW filesystem you are always guaranteed the old file or the new file.
> Maybe you don't know as much about filesystems as you want to appear to?
That is possible, although I'm not sure how much I want to appear to know :P. More seriously, this is extremely condescending. Why is this necessary? If you think I'm wrong, just tell me I'm wrong. There is a pretty decent chance I'm wrong about most things. So, condescension shouldn't be required.
I've had a single issue across many EBS drives partitioned with only XFS, in several mdadm configurations. The data-loss occured during a maintenance window, so I'm convinced it was an EBS issue and not XFS.
I've also used it on physical hardware and never experienced dataloss, even losing drives in RAID configurations (I will admit I was lucky enough to never lose 2 drives in my normal RAID10 setup).
I've had several filesystems become corrupted, but the XFS community is extremely helpful, and the tooling that comes with it has never failed me in at least letting me figure out what's actually wrong.
Basically all of them.
> what are the guides like for setting it up?
Choose "xfs" instead of "ext4" in the installer.
> Doesn't seem like the next-gen filesystem we should be hoping for.
XFS was an early journaling file system. It wasn't designed to be next-gen, it was designed to be fast, service soft-realtime I/O demands and handle large volumes well.
We have seen host kernel panics which damage the filesystem such that xfs_repair cannot fix it. It could be mounted read-only, though I am not sure I would trust the content. We just reformatted and rebuilt content from external sources.
What I believe caused this panic was a habit of creating sparse VM disk images. Later, under heavy write loads from multiple VMs, the host OS filesystem seemed to choke under all the out-of-order allocation needed to place new data into those sparse disk images. When we instituted a policy of always preallocating the VM disk files, we never saw this problem repeated.
That said, it should only cause memory allocation deadlocks, not corruptions.
Is the underlying code terrible? I haven't heard any particular concerns about code quality (aside from the original implementation of RAID 5/6). Most of the issues I've heard about were due to people using features that weren't ready yet. I've heard a lot fewer of these issue since BTRFS started providing clear statuses for individual features[1]. Do you have a different impression?
Its totally anecdotal to say just because my ~10 hard drives with it being stable for two years means its improved but I'd consider putting some faith in it now. A bit. With backups.
It would be great for /home for many normal users - they could always go back. That could also be a problem when one wants to make sure that a sensitive file is really removed. It could be also used at least for some kind of archive. Possibly also for development.
The continuous per-fwrite snapshotting functionality sounds awesome - I've had a few (minor but annoying) accidents with rm of late, so I'd definitely appreciate an rm-proof undelete.
I was trying to figure out if NILFS had any sort of COW functionality, then realized, d'oh, of course it effectively does, via the log.
To address the elephant in the room - checksumming - unfortunately NILFS doesn't (yet?) do this. The WP article links to https://www.spinics.net/lists/linux-nilfs/msg01063.html, which states that the CRC32Cs that are written to disk are used only for the remount/recovery stage, and are apparently not suitable for realtime verification.
Checkout some of the writing about it here: https://www.patreon.com/bcachefs
I'm curious that they're talking about implementing encryption at the filesystem layer. This has typically been done elsewhere. And anyway, key management is an issue. Seems like it would be beyond the scope for a filesystem. But I had similar thoughts right before I started learning about the write anywhere layout used by NetApp all those years ago, where mixing layers had enormous benefits (ZFS too).
The idea of incorporating a flash translation layer (FTL) is interesting, but there is the hardware support issue. Meaning that yes, you can still buy raw NAND flash memory, but what are you going to connect it to? NAND flash controllers which can present a useful interface to the host processor (like USB) already incorporate a FTL.
Similarly, eMMC memory has a FTL layer baked in, and presents itself as a block-addressable device.
Raw NAND flash controllers are increasingly rare on modern microcontrollers.
Personally I'm not a huge fan of this because unlike the clear benefits of giving filesystems control over raw devices, full disk encryption has requirements that can't really be provided partially. You want to ensure no metadata about the filesystem structure or files will be known without knowing the encryption key but this is clearly not possible without having block device layer encryption. ZFS encryption allows for all sorts of useful operations on encrypted filesystems without having the key. This is cool, but also obviously provides a lot of information about the filesystem structure. Also dedup tables aren't encrypted in this setup. ext4 only encrypts filenames and contents, which is worse in some respects.
This sounds like low-hanging fruit for a security researcher to try their hand at: comprehend the dedupe table format, and see what you can derive from it (basically the real-world version of https://xkcd.com/704/).
So an attacker can tell roughly how much metadata you have, but literally nothing about the contents.
I've never heard of "write anywhere" before but it sounds interesting. What is/was this?
> ... incorporating a flash translation layer (FTL) is interesting, but ... what are you going to connect it to? NAND flash controllers ... already incorporate a FTL. ... Raw NAND flash controllers are increasingly rare on modern microcontrollers.
My read of the FTL bit is that of hopeful optimization that supports the ideal-case - say, the filesystem equivalent of all the revving-up everyone is doing to support RISC-V. It's clearly leaning pretty heavily on the bit about existing precedent. My practical hope is that the code stays in good, well-tended condition up to the point it can be usefully taken advantage of - and that someone notices it to good effect sooner rather than later.
[1] https://en.wikipedia.org/wiki/Disk_encryption_theory#XTS
It was a very pleasant experience and I hope that it gets mainlined in the close future. Linux desperately needs ZFS with better integration and more flexibility.
Was it due to bcachefs in some way?
Is there anything you felt was still missing back when you were using it?
> zfs is block based, not extent based, whereas all other modern filesystems have been extent based for years: the reason they did this is that extents plus snapshots are really hard.
then:
>Snapshots: In progress. bcachefs's snapshot implementation is going to be significantly more capable than any competing implementations. When it's done, ...
Multiple ZFS challengers can get easily to the point where they have 80-90% of the cool features that ZFS has. Then everything slows down to crawl because getting the same performance and reliability + features is super hard. When you leave the essential but hard features 'for later' they are harder to add later without big rewrite (or losing performance).
Ticking all the boxes with good performance becomes exponentially harder as number of features grow. There comes time to order things: sacrifice performance in this to attain this.
But even if that's not true, bcachefs still has an excellent feature set already with great performance, especially in long-tail latency.
He or any other excellent guy don't have an idea how the implementation he designs works in multiple workloads and different machines and disk drives. There will be time period of several years of adjusting and tinkering and compromising.
the appeal of doing raid5/6 within a COW filesystem is that it's potentially an opportunity to get rid of the write hole. Of course, the devil is in the details.
Btrfs was a known bug, and the kind folks on #btrfs were able to help me recover (though it was a quite involved process).
For ZFS, the consensus in #zfs and #zfsonlinux was that with the error message I was getting "I hope you have backups."[1]
I see XFS mentioned here, but in the admittedly unusaly use-case of editing a file that when compiled and modprobed may hang the kernel, I found that vim was insufficiently fsync-ing to ensure I had valid data (either old or new copy would have been acceptible).
I don't know what my point was with this comment except "All filesystems are buggy, so backup your data"
1: It should be noted that in the case of ZFS, I was on vacation and it was my laptop that had no critical data, but I absolutely needed it working for other things, so I only spent a couple hours trying to recover; The btrfs issue happened at home where I had plenty other systems to use and where I was more motivated to recover the data.
People who have been running btrfs in production (reckless, imho) on kernels that are 2.5+ years old or worse. And then they're surprised and disappointed when it loses data.
For everyone who's posted here with a btrfs data loss story, it would be helpful to quantify your anecdote with what kernel you were running.
As far as why, https://code.fb.com/open-source/linux/ has a nice quote:
"Btrfs has played a role in increasing efficiency and resource utilization in Facebook’s data centers in a number of different applications. Recently, Btrfs helped eliminate priority inversions caused by the journaling behavior of the previous filesystem, when used for I/O control with cgroup2 (described below). Btrfs is the only filesystem implementation that currently works with resource isolation, and it’s now deployed on millions of servers, driving significant efficiency gains."
(I don't know though)
It's very common for MegaTechCos to hire maintainers like this, but us in the hoi polloi need to be cautious not to jump headfirst into a situation where we can't call upon the experts/inventors to get us out of any jam. We have to rely on our own abilities, the mailing list archives, and hopefully some relevant random blog posts that aren't too outdated.
This should be taken into consideration much more often than it is. If anything, maintainer employment by a MegaTechCo should nix out their implied endorsement of the technology rather than taken as proof that it's viable.
In the case of OpenZFS, where it's implemented by multiple operating systems with differing requirements, maybe the different priorities and goals of the different teams will result in cross-pollination of ideas and implementation details rather than it stagnating top-down. I.e. there's scope for some differences in features and details, while still adhering to the common design.
I'm a bit confused by this. Irem wrote:
> But if Debian decides to drop x86 tomorrow, you go "Cool, good luck, I'll just fork."
When Systemd made its way into debian, debian _was_ forked by devuan (and possibly by others) to keep it out. So...is your point exactly?
edit: Maybe I misunderstood entirely and you were using Systemd as an example _supporting_ Irem's post...
There are areas where it just doesn't work, such as in the embedded space.
Maintaining an entire Debian architecture is probably not a one-person job.
Whereas Apple really could just up and drop x86 any day now, and I'll just be stuck.
My current backup server would need ~50GB of RAM just for the in-RAM deduplication tables.
I've been thinking of trying btrfs for the backup server, but the deduplication story is not something that's obvious, it all seems to be third-party, which makes me skeptical. But it worth trying. I might just try a test Dragonfly setup, because my simple testing of HAMMER seemed very promising, years ago.
I've had pretty bad experiences with btrfs in the distant past. Around a decade ago I had all my company laptops using btrfs and it worked great, as long as we didn't get >80% usage or something. It worked well for around a year. But then the reliability of btrfs dropped, and we had repeated data loss and switched back to ext.
btrfs wasn't considered stable until 2012. Using an experimental filesystem for all your company laptops seems to be asking for trouble.
There are some other types of dedup that don't guarantee 100% dedup, but then the dedup levels would very depending on data sets -- this may be fine for VMs and user data. They work by being using data locality, predicting which dedup table would needed to dedup the next N bytes. But it is not likely we are going to see this any time soon because the patents on this sort of dedup is fairly recent.
This is only true for real-time, "live" dedup, which is an essential feature for almost nothing.
btrfs' offline dedup (which, despite the name, can be done on a mounted filesystem) is far more rational. You fill the disk with data, and then dedup it later, at a schedule that works for you. No extra RAM needed.
You still have to process it, and you are still going to use a lot of ram -- and you are going to want to use a lot of ram once you get into larger data sets otherwise the process will take days.
That seems only slightly less scary than the btrfs on Windows project (which suggests disabling SecureBoot). At least it's from a ZFS contributor (while btrfs on Windows seems like a from-scratch rewrite, ie. all the bugs all over again).
From the comments here I take it people aren't actually using anything with features like snapshots/cow/compression on Windows :-/
Microsoft has somewhat ridiculous driver signing requirements. I become angry about this perhaps once every four months.
FAT32 "seamless support"? It doesn't even have journaling. The other day I wanted to copy a 8 GB file. Forget FAT32. Just use Ext4. It isn't patent encumbered, and there are drivers available for all of the big 3. You can boot a live CD/DVD/USB to mount the data in no time. This is easier than installing any driver, and you won't have to uninstall or maintain it afterwards.
It's better than FAT and widely supported.
https://github.com/maharmstone/btrfs
(I have never used it, just have seen this refered in the ReactOS release announcement...)
It’s all network. So all I need is working file-transfer protocols.
I feel like so much of BTRFS though was death by committee. It was trying to do every single thing ZFS did, regardless of whether or not the specific feature from ZFS was mediocre to begin with.
focusing on a release cycle and communicating would have helped it alot instead of vague things like "it is ready enough" and "some features arent ready"
Those customers can't redistribute Solaris any more than they can redistribute Linux with a CDDL-licensed ZFS. If they really wanted to switch to Linux they could do so today and be in the exact same legal position license-wise. Not to mention they could switch to FreeBSD and be able to redistribute, but again, I doubt this is something very many Solaris users care about. Perhaps the cost of porting their applications away from Solaris is too high, or their middle managers are Oracle fanboys, whatever the reason is, a GPL-compatible ZFS does not threaten that business model.
I think a more likely reason why they pour money into the Btrfs death march while having a turnkey solution just sitting around in the next office over waiting to be singed off by the legal department as CDDL+GPL or Apache or whatever is that each project is the pet of a different pointy-haired boss and they refuse to cooperate internally.
I run exclusively with mirrored vdev’s and have never run into issues growing my pool.
Perhaps you're thinking of shrinking, which it still cannot do without rebuilding the whole FS infrastructure.
My opinion after trying both parity-checked redundancy and true mirrored redundancy is that any serious data storage should be strictly on mirrored volumes (preferably with disks of substantially different ages so you don't risk them dying within a couple weeks of each other, possibly before a replacement gets installed).
I once read "50% storage efficiency is plenty, don't be greedy". Although painful for the home user, that seems like the right attitude to take for important stuff, especially as volume sizes increase. It can be a very slow and very fraught process to recover several TBs from parity data.
I've been using it on a 6 drive NAS since 2012, and on two laptops with SSDs.
In my experience "crashes" are symptoms of people who didn't read the documentation and don't understand how to recover.
For example, on the laptops I've often run out of disk space. And btrfs can run into a situation where it can't clean up partly used blocks without more disk space. To recover you have to add more space. I usually used the swap partition, but a USB stick also works. Or a RAM drive, but that's risky.
Some other issues I've had to help people with is snapshot deletion. Deleting a btrfs snapshot appears instant but none of the used space comes back. That's because it is doing it in the background and it can take a really long time. OTOH, ext4 does exactly the same thing with background delete of very large files.
aka most users. Filesystems must be reasonably resilient to incorrect usage. This is a problem across the board; if you're not careful setting up a ZFS pool, it's easy to accidentally create a scenario where you're unwittingly striping across multiple devices, greatly increasing the likelihood of a catastrophic data loss event without ensuring the user is fully aware.
Yeah that's every filesystem ever. I helped a guy out just a couple of months ago because his Debian virtual machine had plenty of disk space but he couldn't install anything.
Because it was out of inodes. What "regular user" aka Nodejs / React developer knows about inodes? Not that guy.
So I guess ext4 is junk too.
I don't use RAID5, which is where I've heard btrfs is dangerous, just for the snapshots and dedupe (with https://github.com/jbruchon/jdupes/ )
(I work at fb)
That video's discussing btrfs layering, which is very neat.
I am _very_ curious what sort of experience FB has had with btrfs.
If FB has decided to go all-in with btrfs, you probably have the most accurate raw data there is to have.
Of course, any hyper-scale deployment of a technology is going to produce remarkable/exponential statistics, and these will (sadly/annoyingly) need careful parsing to normalize in a way that is generally accessible and unsurprising. (We live in a very knee-jerk world, and all. Sigh)
All this to say, I at least am looking forward to whatever sorts of numbers you end up being able to share.
--
After considering the bit about btrfs in the video, and your mention of "1 or more machines", I wonder if you aren't using btrfs on things like load balancer type systems - or, more abstractly, node configurations that can always safely be "thrown away", whether by hardware failure or explicit decommission. But then I realize the chances are you probably use such a model (individual hardware failure must be acceptable; everything important must be n-way redundant) because nothing else makes sense at scale. And then I wonder... exactly what point in that spectrum does btrfs fit in? I wonder if/how you can answer that question.
Doesn't appear to be merged yet, but it's coming.
- RAID5/6 mode is only experimental in btrfs, while it is really stable in ZFS.
- I don't have concrete data for that, but in my experience, BTRFS has high latency (>1 second) even for small file operations when under load.
- While that should not be a problem for production systems, I have some crappy hardware where BTRFS oopses or corrupts data, while other filesystems (Ext4, ZFS) work fine.
I even had to add eatmydata support to the schroot tool as a command-prefix option to allow every command to be run via eatmydata when using btrfs snapshots. (I've since dropped btrfs support entirely; it was too unreliable with large snapshot turnover rapidly unbalancing the filesystem. Unusable in production.)
When there's only a single filesystem, and that filesystem is btrfs (or ZFS), it should however be possible to optimise this away and delegate everything to the filesystem. But even here, maintainer scripts may issue their own fsyncs as they update their own databases, kernel images or whatever.
Not if file-change notifications were supported robustly by dpkg and the kernel (to a lesser extent). Getting to that would, however, require massively restricting the compatible-kernel-versions set of dpkg, and would also probably require undoing some of the more . . . misguided pieces of history with regard to file-change notification systems in Linux.
The problem is that the system state needs checkpointing for every package state change. It must allow for recovery on failure, termination, abortion or power loss, amongst other scenarios. And the package database must remain in sync with the filesystem state.
When every managed file is on one snapshot-able filesystem, this could be rolled back atomically, and the fsyncs skipped. But as soon as you have a non-snapshot-able filesystem or multiple filesystems in use, the fsyncs can't be skipped.
Had there not been uncertainties about licensing which prevented ZFS from being in mainline there wouldn't be any Btrfs. How can you ever really trust a filesystem developed by adding features first, trying to add stability and usability later anyway?
I did think it was odd that installing the ZFS package took such a long time, but that was a one-time wait.
While working in that specific industry it seems like there is a motivation to specifically go out of their way to make things unreliable, extra complicated and utilize bad technologies.
The fact that IBM and Toshiba on POS is frightening.
in addition, we know that there are patent landmines out there because NetApp and Oracle have a (undisclosed) settlement on the issue. Using ZFS is a probably good way to get a visit from your legendary friendly Oracle sales agent and auditor. Oracle could fix all of this with a quick re-license or patent indemnification, but they haven't and almost certainly won't.
My Synology (and all modern Synologies) uses BTRFS, and it seems pretty solid so far. I would have preferred that ZFS had matured and been dominate in FS, but there are some really risky things that come with ZFS.
The CDDL also includes a patent agreement, Oracle can’t retroactively waive it.
Go look at what the Java community is going through right now.
You did not understand what happened with the latest Oracle Java news and you are spreading FUD.
People have a choice what Java implementation to use, they can pay for latest Oracle implementations, use Open JDK (it is included in Debian and most distributions), you can get support from RedHat and other big vendors. There is no risk of getting in trouble by using Open JDK, MS and Google had trouble because they "forked" Java
Thanks for letting us know one nerd's take on openzfs.
The only thing standing in zfs way (to 1st tier Linux support) is a company springing up and heading the charge. 75 percent sure this will happen eventually with ZoL at some point in a decade's time.
However, another point of measure is money. Synology is huge and seems to think btrfs is good enough for business. They compete with ixsystems directly. We could go all day and into 2020 with points and counter points. The licensing issue is weak but will eventually be moot with enough money and time behind the right project. I'm not in any position to judge that last point for sure just offering my IT pro/business take on it.
To be fair bsd-ish-ness has a role to play on the backend and is bullet proof and battle tested for the right mission. But the same can be said for Linux too. The last point I'll make is that today, I can run the Linux kernel from my msft windows 10 machine natively. No one here would have saw that in 2000. Does bsd, zfs have a similar anecdotal Hero Epic like this; where the enemy capitulated so thoroughly?
% findmnt /
TARGET SOURCE FSTYPE OPTIONS
/ rpool/ROOT/default zfs rw,relatime,xattr,noacl
The main missing piece is proper support in the installer for using it as a root filesystem. Currently this requires manual setup, which is tedious, but even this is functional once done.
The special thing about the "Linux on Windows" subsystem is just how different the host OS is, which requires Microsoft to do some quite extensive changes to the Windows kernel in order to support it. Unix to Unix is relatively easy by comparison.
OTOH, we also have Wine, which is pretty much exactly the same sort of thing as Windows 10's Linux subsystem.
another point that shows that ZFS is unsuitable for the desktop is it's reliance on EEC RAM: https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-...
i considered getting EEC RAM for my machine just because of ZFS, but i could not find any 16GB strips to fill my board.
that's two strikes against desktop use, and enough to let me prefer btrfs.
EDIT: i am very happy to see so many replies proving me wrong. when i last researched this topic i was not able to find any counter arguments to needing EEC.
greetings, eMBee.
Edit: oh and it works beautifully with single disk setups too.
That's like saying you don't need health insurance because you haven't got ill.
Claim that you can only use ZFS with ECC memory to feel safe is false.
I used to use non-ECC memory on my laptop when running Belenix, and I now use ECC memory on my servers at Hetzner where I run my own Illumos builds.
(I'm one of the maintainers of Belenix, an early opensolaris based distro that was the foundation of Solaris 11. I'm working on reviving it again)
“There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. Actually, ZFS can mitigate this risk to some degree if you enable the unsupported ZFS_DEBUG_MODIFY flag (zfs_flags=0x10). This will checksum the data while at rest in memory, and verify it before writing to disk, thus reducing the window of vulnerability from a memory error.
I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS.”
https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...
W.r.t. to btrfs they are at least working on a fsck tool, even though it can't repair much yet (what it can repair is e.g. some bitflips caused by bad RAM: https://patchwork.kernel.org/patch/4116411/ ).
And stuff like ext4 or ntfs can get back to a state where one can write to it again in most cases.
Another problem is that many applications don't handle EIO well, i.e. they don't give you an option like "Continue with corrupt data", e.g. if a video file gets damaged it doesn't matter much, it'll just be a few damaged pixels, but the whole video player stops playback because it doesn't handle the IO error.
If you read closely the posts that are supposed to prove this claim, they never give strong evidence that the pool losses are directly linked to the lack of ECC memory.
It always ends up with cyberjoke stating that the issue was linked to lack of ECC.
Moreover, One of the OpenZFS developers confirms this is a myth here : https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...
TL;DR: Zfs is as vulnerable as any other FS to bit flips. So if you care about your data, use ECC, regardless of your file system.
TL;DR is that a few stuck bits is won't corrupt your pool like the forum post claims.
Also, in the highly unlikely event that bad RAM would make ZFS corrupt data, how would btrfs perform better on the same hardware?