Hacker News Reader

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.

My experience has been aggravated by being simultaneously affected by https://github.com/openzfs/zfs/issues/11893, which results in all zfs operations hanging (kill -9 is ineffective) until a power cycle. This round of upgrades was not fun.

Debian bookworm users: You need to enable backports for zfs. Both bugs are fixed in backports but still present in stable.

Let's do away with the tribalism in both ways? Calling it perfect is quite a stretch. I'm not saying "don't use zfs", just "you want to be prepared for the worst-case if your data matters". 3-2-1 and "validate your backups" still apply.

Perfect for me is not perfect for others of course. I have not run into the issue you linked, that sounds frustrating to deal with.

Backups is the only (near) 100% way to insure you data I agree.

As someone currently rewriting Ansible plays to build Bookworm QEMU images with ZFS support, thank you for this incredibly niche and useful bit of information.

Np! Actually, in case you're able to share some of those plays in any form, that could be very useful here, however niche :)

I have an older version [0] for Bullseye that you're welcome to take a look at. It's actually combined Ansible + Packer, so if you point it at a Proxmox host, you get an image in ~10-15 minutes.

It's a fork of other work [1], modified and updated for Debian and Debian only. It also in absolutely no way has any best practices, which is why I'm doing a complete re-write with proper role dependencies, modularity, etc. When that's done, I'll be marking [0] as archived and linking to the new repo.

The new version also replaces the incredibly hacky shell script builder (hooray dialog) with Python, and updates Packer's JSON with HCL2.

[0]: https://github.com/stephanGarland/packer-proxmox-templates

[1]: https://github.com/chriswayg/packer-proxmox-templates

Thank you!

No, you're just hearing about ZFS corruption like you've heard corruption in other filesystems, this wont be the first and last.

Its easy to feel like its being targeted with some kind of campaign, however the truth is rarely that exciting. The software is being used by more people, which means it will expose more bugs.

No software is perfect, it just now starting to be abused enough to be important enough to be talked about. This stage is entirely normal.

But it's totally different than corruption in other filesystems. People are acting like upgrading to ZFS 2.2.0 ate all their data like XFS used to back in the bad old days. I remember once the power went out at my house, and the entire XFS filesystem was irreparably damaged.

This bug is super hard to trigger, and has roots back to 2006 and only surfaced with a recent coreutils update. Yet there are numerous posts in the past few days verging on hysteria. I myself was very worried until I saw posts like the grand parent comment or the linked GitHub issues.

I don't believe there's actually a campaign of ZFS haters sitting around in a storeroom somewhere posting to phoronix and HN, but it does feel like a lot of strongly held opinions are being unleashed now that ZFS had this bug.

I'm pretty sure ZFS has been abused a lot - ten years ago I was helping test petabyte scale ZFS clusters, doing tons of concurrent IO and such. I'm pretty sure Sun and Oracle tested the crap out of it.

> But it's totally different than corruption in other

> filesystems. People are acting like upgrading to ZFS 2.2.0

> ate all their data like XFS used to back in the bad old

> days. I remember once the power went out at my house, and

> the entire XFS filesystem was irreparably damaged.

Different bugs manifest differently due to the coding, this is just the same.

ZFS wouldn't even have a hundredth of the testing of real-world use ext3/ext4 in hours. The large scale deployments of ext3/ext4 dwarf the usage of ZFS.

ext4 released: 2008

https://archive.ph/20120529150649/http://git.kernel.org/?p=l...

ZFS released: 2005

https://web.archive.org/web/20130619165135/https://blogs.ora...

This is Linux local bias. There is more to Unix than Linux.

Back around 2001 when work on ZFS began, I very strongly suspect that there were several orders of magnitude more storage on ZFS on Solaris in datacentres around the world than there were on all Linux filesystems combined.

Just because you are more familiar with the FOSS tool doesn't mean that it's more mature or more widely used or more deployed in production.

I would -very- strongly imagine it the other way. I guess we'll never know.

I worked at a company that shipped petabytes of ZFS storage for years and years. Think 42u racks with 60 drive JBODs. Hundreds of these. Hundreds. And they were a very small company. Can you imagine the scale Sun and Oracle shipped hardware at? Was there any linux based hardware company operating at the same scale?

I don’t think we had any customers request ext4 bulk data storage. OS drives sure. Everything else was ZFS. Or weirder proprietary filesystems!

Yeah, that GPFS was real pita, i'm not talking about size comparison issues. I'm talking about runtime in the field.

You simply can't begin to understand the amount of shitty iot and android devices that exist.

See the earlier comment to you.

I think you have this 100% backwards. I heard similar comments to yours inside some large Linux vendors, too, and I think they betray a profound misunderstanding about the nature of large-scale Unix deployments over the years.

FOSS Unix is recent and mostly small-scale and transient deployments of containers, not long-term storage... but before that, there were decades of vast scale commercial Unix systems, which dwarf the FOSS stuff.

How many zfs systems you think are running right now, all hours combined compared to linux ext filesystems ?

"Bananas are better than passion fruit because there are more bananas being grown!"

ZFS is an enterprise storage-management tool. It's not just another hard disk format, like ext4.

Not many people are running Linux boxes with root on ZFS because few distros support that.

Single-disk setups are not its strength so the number of single-disk setups of ZFS compared to ext4, which only supports single-disk setups, is irrelevant.

Even saying that, though...

I bet all FreeBSD deployments are on ZFS, and it's got a solid enterprise presence. Anything streamed off Netflix anywhere is being served by FreeBSD.

Oracle still supports Solaris. All Solaris, OpenSolaris, Illumos, Nexenta, OmniOPS, SmartOS, Tribblix... all on ZFS.

And it's still not its core area.

If you want to do a fair comparison let's hear about companies doing petascale to exascale arrays with double-digit to quadruple-digit numbers of drives.

How many of them do you reckon use ext4?

I'm not the one who moved the goal posts here though was I ?

Here, let me quote myself.

> The software is being used by more people, which means it will expose more bugs.

ZFS and ext4 is so completely different on a fundamental level that there is zero reason comparing them, it just shows your "expertise" on the topic..

I dont think it takes expertise to do basic math. But aint no value in ego measuring online.

To me it does feel like this has been on HN a bit more than other filesystem corruption bugs. This bug can basically only be triggered by using lseek to search for a hole. Ext4 has a very similar bug, but that requires enabling inline_data (and possibly a non-default blocksize, but that doesn't always seem to be the case): https://forums.gentoo.org/viewtopic-t-1166006.html

I don't think I ever heard about this, apart from in the context of the ZFS bug. And although inline_data is niche, ext4 as a whole I would argue is not.

Actually lseek seems to have been broken on most filesystems at some point: https://bugs.gentoo.org/891125 https://github.com/gluster/glusterfs/issues/894

And apparently apart from modern coreutils using that, it is mostly gentoo users hitting the bugs in lseek.

I'm unreasonably annoyed you used etcetera instead of et cetera.

'Et' is Latin for 'and'. The term means "and other things", not "andotherthings".

No, I have nothing important to add to the ZFS discussion.

etcetera is found in ~500 years old books, and et cetera itself comes from et caetera, from the Greek καὶ τὰ ἕτερα. Which one should we use then? Should we only use the Latin from 1 BCE/CE or the vulgar Latin that evolved after? :-)

Maybe I do not feel it like a big issue, since many Latin languages use it a single word (such as Italian with eccetera, Spanish, Portuguese), due to the evolution of the language.

I guess even ~500 year old books can be wrong :-)

I'm just joking. This is just a pet peeve of mine. Thank you for the fun lesson in etymology.

Did not know this, I don’t mind learning something. You’s think though that autocorrect would pick up on that…

The reason, and the difference, is that all these other filesystems have check and repair (and sometimes multiple) tools.

Please correct me, but ZFS has none.

What good would ZFS be without that?

The key differences are in two places:

1. Every administration task for ZFS is done online. You don't need to take your pool offline just to check for errors and repair them (if possible). (Mind that on FreeBSD, UFS can usually have a fsck done in the background while the file system is in-use. Just about zero Linux-native file systems have this capability)

2. "other filesytems" can only hope to detect and repair inconsistencies in their metadata structures. If your files are corrupted, they can't know and won't tell you that they are. ZFS checksums everything, including regular file data. It will repair regular file data too, if possible.

You're completely wrong.

ZFS's scrub is both a check and a repair tool. It's already saved some of my data.

You are partly right. Zfs scrub is a repair tool when it has parity / mirrored copy of data to recreate it.

It's safe to make the assumption that the tool isn't magic.

A scrub can also repair data when using `ncopies` greater than one even outside any mirrors or parity.

Zfs has checksumming and scrubs, which can catch lots of data corruption that (most?) other filesystems can't catch catch at all

I guess I meant it as having a separate repair tool, but yes, you're of course correct.

To me it was always weird to not have a separate tool for being able to do an offline ZFS repair.

With regards to data corruption, I mean, this is exactly why I moved to btrfs, because it's able to catch bit rot same as ZFS.

I think ultimately this is why ZFS data corruption bugs always are such a big deal... It's because in a lot of tech circles ZFS is put on this infallible pedestal where it can never do any harm to your data.

> To me it was always weird to not have a separate tool for being able to do an offline ZFS repair.

Have you ever asked yourself why those operations need to be done offline?

It's not that ZFS doesn't have these capabilities, it's that ZFS is doing these similar checks and repairs as part of its normal operation and more. If you can't mount a ZFS filesystem, odds are a similarly borked ext4 filesystem isn't going to be repaired by fsck.

I got bit by some pretty bad foot guns in BTRFS that has meant I steer pretty far away, but my understanding is these have been somewhat mitigated. I hope it continues to work well for you.

Could you please explain what you mean by a separate repair tool kit?

I think he means an equivalent for `fsck`.

`zfs scrub` is equivalent to `fsck`.

It just has a different name, and it can fix more kinds of corruption (including corrupted data, which `fsck` for other file systems can never do), and can do so on line. Of course zfs can fix those same errors on line while doing ordinary reads, so really all `zfs scrub` does is read everything.

`zfs scrub` is better than `fsck`.

It is not.

`zpool scrub` walks every block in the pool, and implicitly, you do checksums and other things while doing that. That's it.

It's not doing any sort of logic bug repairs or cleanup or anything else.

It's also not checking that you can, say, decrypt things, since that would mean you needed the keys to scrub.

It really is better. The key difference to understand is that in ext4 any form of metadata corruption which can be automatically fixed requires you to take the filesystem off–line (to unmount it), and then run fsck. Meanwhile with zfs, any form of metadata corruption which can automatically be fixed is simply fixed right on the spot, transparently.

In truth fsck is a wart, a kludge, a bag on the file system design.

I am passingly acquainted with ZFS.

The key thing to realize is that you can't, actually, automatically fix every problem, sometimes you have found a logic problem which results in an impossible outcome and you need someone to manually clean it up.

In a world without flaws, it would be great to never need that. But the thing about theory and practice is that in theory, they never differ, but in practice...

I know that. But fsck only fixes the ones that it can fix automatically. With anything else, you are completely on your own. Get a hex editor and go to town. With zfs, if there is some kind of problem that cannot be automatically fixed then at least you have one more tool available: zdb. It’s a _debugger_ for zfs filesystems. It will show you everything, more than you ever wanted to know. It is way better for fixing problems than any hex editor.

zdb is read-only. It's not fixing anything, just telling you what's going on.

Don’t be an idiot. You can fix more with zdb and a hex editor than you can with the hex editor alone.

That seems rather rude.

The discussion was about the need for tools to make it easier to handle cases where you couldn't automatically handle repairing them, and your statement was that zdb is very useful, which is true, but it doesn't fix anything.

fsck for various filesystems has a bunch of common cases like "this is an orphaned file, should I save it or mark it free?", and something similar would indeed be useful for a number of failure cases in ZFS which require more explicit instruction on what to do about it because you can't easily automatically resolve it.

`zpool scrub` is very useful, but ZFS could still benefit from automated tooling to handle some common failure modes, not just let you write bespoke tooling every time.

> fsck for various filesystems has a bunch of common cases like "this is an orphaned file, should I save it or mark it free?"

This is an example of an automated fix that zfs just handles transparently, without needing to prompt the user. Or it would, if zfs could even have orphaned files, which it cannot.

I don’t know why you think this is such a win for fsck, which doesn’t even bother to give you any idea what the file was. It doesn’t try to show you the contents, and it probably doesn’t know what the file was called, or why it was deleted. Or even if it _was_ deleted; a file could be orphaned merely because the data was written but the write to the directory entry got lost. The user has nothing to go on and just guesses, or says `y` for everything. Useless.

> `zpool scrub` is very useful, but ZFS could still benefit from automated tooling to handle some common failure modes

This is precisely and exactly what scrubbing does! All failure modes that can be automatically fixed, whether they are common or not, are transparently fixed without even needing to unmount anything.

zdb is there for the really rare cases where there is so much damage to the filesystem that zfs cannot even mount it safely. Other filesystems don’t have anything like it.

> Or it would, if zfs could even have orphaned files, which it cannot.

If you use the zfs_recover parameter, then in a couple of cases, it will just permanently mark space as unallocatable forever because it can't figure out what owns it due to some errors, and you decided that was a better outcome than whatever error it was encountering. (That's what the "leaked" zpool property counts.)

Conceivably, you could write something to carefully figure out what, if anything, owns it, and allow it to be freed, or grab the contents of that region and drop it somewhere and then throw it out after the same safeguards, but by definition if you triggered that handling, something has gone wrong and you don't have a better automated intervention.

I wasn't arguing that ZFS has a case requiring orphaned files, but my point was that that was an example of "I don't know what to do with this, but I know enough to realize I can't decide what to do about this or just throw it out, so here, you do it."

Or if you, for example, wanted to do a destructive rewind on a pool because of some horrible edge case, then you might want some automated way to extract everything you're about to throw out, and that might require more work than just `zdb -R` if the pool won't import in the first place.

One example of something that would be useful to be able to do, particularly offline, would be when pools have some issue like spacemap corruption, you could conceivably walk the non-spacemap metadata to re-synthesize the spacemaps from whole cloth and write them out again. (And it's not an all-or-nothing thing, a number of versions many years ago would have very minor errors in how they computed spacemaps, which just mean zdb whines at you if you ask it to verify them even when the pool is offline, but don't interfere with the running otherwise.)

Could you convince the kernel code to do that for you on import if it's blocking import? Probably, though you'd probably want some out of band communication method to force it sometimes because I would bet there's any number of ways it might go awry and get too far into the woods of thinking it's "fine" before tripping some assertion.

Is it going to be faster to iterate on it from userland, particularly if you can simulate whether the import works from something like zdb with those conceptual spacemaps written out somewhere, versus rebooting on kernel panic? Absolutely.

But that's an example of a case you might have something so disgruntled it could be nice to reconstruct it outside of the normal import flow.

> All failure modes that can be automatically fixed, whether they are common or not, are transparently fixed without even needing to unmount anything.

`zpool scrub` doesn't fix anything except checksum errors. That's it. Any other class of flaw, it doesn't handle. I don't know why you think it does anything else, but I promise you, it absolutely does not.

> ask yourself if you've ever had highly parallel workloads that involve writing and seeking the same files at the same moment.

Uhhhh, databases?

Databases don't involve using SEEK_HOLE to find gaps in a sparse file, usually, so it wouldn't come up here.

So you needn't do so: SEEK_HOLE does not occur in the GitHub.com repos for postgresql, mariadb, nor sqlite. Are there system libraries they incorporate which use SEEK_HOLE?

>> So while I would never say that no one has ever hit the problem unknowingly, I feel pretty confident that they haven't. And if you're not sure, ask yourself if you've ever had highly parallel workloads that involve writing and seeking the same files at the same moment.

It makes it sound unlikely, but if I have a couple of VMs in datasets (all formatted as ext4 internally and some running DBs inside them) each is one big `raw` file which is getting a lot of reads and writes, I assume.

How are they at risk?

Also, what about ZVOLs mounted as ext4 drives in these VMs?

This is a great explanation, thank you.