Hacker News Reader

Comment by GauntletWizard | Hacker News Reader

As someone who is heavily dependent on zfs, this is a little bit of reassurance. Sad that data corruption exists in any file system that is shipped too many people, but reassuring that it happens to the most stable and least "interesting" in the newness sense file systems.

ZFS's own recent file system corruption issue is in roughly the same category of edge case, but accessible to reasonable if niche workloads.

View on HN

Note though that unlike zfs, ext4 has a battle-tested fsck.

Combine ext4's dumb but robust approach to journaling and robust metadata layout (for example inodes are statically allocated) with fsck.ext4 (which got refined for years) and you can recover from any situation.

To give you an example, fsck.ext4 will happily carve a working filesystem out of random data, as long as there is a valid superblock. Seriously - try it yourself:

  # create a working filesystem image
  dd if=/dev/zero of=test1.img bs=1M count=256
  /sbin/mkfs.ext4 test1.img
  # write file with random data
  dd if=/dev/urandom of=test2.img bs=1M count=256
  # copy superblock into random file
  dd if=test1.img of=test2.img bs=1024 count=4 seek=1 skip=1 conv=notrunc
  /sbin/fsck.ext4 -fy test2.img
  sudo mount test2.img /mnt/somewhere

I'm trying to understand what exactly this is testing/demonstrating.

Like: "Wow, but why should I care?" I'm not sure being able to fsck a random disk image shows resiliency. Doing this could do all sorts of nasty things to data you actually care about.

(not the author) The metadata is a contiguous range of disk blocks. I think the intuition is that such layouts are likely to require simpler filesystem code to manipulate. [versus (iirc) ZFS which may have metadata blocks scattered throughout the disk, probably requiring more intricate code to keep track of].

> [versus (iirc) ZFS which may have metadata blocks scattered throughout the disk, probably requiring more intricate code to keep track of].

So -- you think redundant metadata is a bad thing? Try wiping your metadata and then trying to fsck that random disk image.

Again, has this been the source of data corruption you're aware of? This seems like a "Maybe, it could be this way, but I don't know" kind of take.

> To give you an example, fsck.ext4 will happily carve a working filesystem out of random data

What application does this have for recovering user data? Would it not be more interesting to see how much of the user data can be recovered? Or are you implying that the random data can symbolize the user data here and much of it would be recovered?

You’re forgetting the btrfs folks.

— someone who lost data due to btrfs bug

Yeah, all the threads full of folks bashing on ZFS because their trusty ext won’t corrupt data should be eating crow right about now, but I’m sure they’ll hop in and explain why this bug is different.

It's not an extfs bug, it's a "stable kernel" process bug. Linux master branch/mainline was never affected by this bug. Someone cherry-picked a wrong patch into stable without realizing the consequences, and it was caught too late.

Stable process is cherry pincking thousands to tens of thousands of patches from the current master kernel branch into years old kernel branches, spraying tens of thousands of emails at original patch authors, hoping (With some limited testing on top) that the resulting frankenkernels will still work and all these patches will have satisfied dependencies applied, too.

I don't trust this process very much, and just run the latest stable branch for the latest kernel release, only. Staying with older stable release branch for too long seems too risky, unless you're some bigcorp that can afford the testing required, or you're running some highly mainstream setup that is probably covered by tests done by the stable team, and testing teams they cooperate with.

I understand your reasoning, but running a popular « stable » distro in a popular configuration will help detect bugs much faster. I’m not sure this would be the case for the very latest kernel.

A bunch of popular distros run the latest kernel.

Nobody ships vanilla kernels. Sure the reference implementation was unaffected but users never experience vanilla Linux kernels

Distros like Arch and NixOS use the vanilla kernel with very small amount (single digits) of patches on top. For the purposes of ext4, it's very likely to be as-is with vanilla. (I am counting the official linux-stable releases from kernel.org as 'vanilla' here.)

Fedora, openSUSE tumbleweed, Manjaro,...

> Fedora

You know it's not that hard to debunk this:

> You'll then be left with a kernel-6.X.? directory, containing both an unpatched 'vanilla-6.X.?' dir, and a linux-6.X.?-noarch hardlinked dir which has the Fedora patches applied. [1]

WOW fedora patches! Sounds to me like they're not shipping vanilla.

[1] https://fedoraproject.org/wiki/Kernel

That's not what my original argument was about. So not sure what you're debunking.

I have problem with backaptching 10s of thousands of changes to years old kernels by people who don't really understand the changes or consequences, like 5.4, 5.15, 6.1 or whatever. Not patching up 6.6 or 6.7 kernel with a few out of tree patches, where they mostly understand what they're doing and can test the limited set of changes they're applying.

If they change anything it's no longer vanilla. Anything at all. No matter how insignificant.

Even a single patch to something like default kernel config parameters means it's not vanilla. I'm not arguing that there are significant patches, only that nobody seems to ship a true vanilla kernel.

> Even a single patch to something like default kernel config parameters means it's not vanilla.

Anybody who cares that much is very likely already compiling their own kernels (I speak for myself here). It doesn't make sense for distros to do the extra work to support it.

Just expanding on this a bit: my Debian laptop has a Kconfig with modules disabled that only includes the exact set of drivers it needs. It takes ten minutes to rebuild on the laptop when I pull from git. Even if Debian did all the work to let me automatically install the latest vanilla kernel... I'd still build it myself.

This isn't a sports tournament, nobody is cheering or bashing "teams". If ZFS has bugs and caveats, they should be called out. Same for ext4.

The problem here was Debian's distribution process. Any distro compiling linux from the releases on kernel.org was not affected.

So this is another bug introduced by Debian itself by patching things?

I remember that there was a fairly severe one which was caused by patching OpenSSL I think? But I remember the change they made being fairly weird and no one understood why but it was easy to see that it would introduce a vulnerability.

No, the issue was caused by backporting a patch A but not backporting a patch B. Sadly in this case the overall behavior after applying just A was broken when issuing direct IO writes.

I remember when they made 2 CVEs back to back in Apache by incorrect back porting of changes

No, it exists in the upstream release.

Are you sure about that (genuine question)? The linked discussion involves a Suse engineer and a request to the kernel maintainer directly, not to a Debian packager-

No, its an upstream bug being discussed in the debian bug tracker

This is a misreading of the bug. It is from upstream stable kernels before 6.5 that include commit 91562895f803 but not 936e114a245b6[1].

In this case Debian's current process is good - it's kernels track kernel.org stable releases. This debian bug is responsibly flagging "for visibility" that a serious bug has been discussed and fixed upstream.

[1] https://lore.kernel.org/stable/20231205122122.dfhhoaswsfscuh...

Do you have a link to just the 2 patches (in patch form) that could be before on a kernel tarball to revert these 2 changes?

This is not accurate. As you can see in the changelogs, the problematic commit made it to 6.1.64 and the fix was merged in 6.1.66:

"properly sync file size update after O_SYNC direct IO": https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.6...

"update ki_pos a little later in iomap_dio_complete": https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.6...

This post explains the relationship between the two commits: https://lore.kernel.org/stable/20231205122122.dfhhoaswsfscuh...

What if you build your own kernel from kernel.org, but you use Debian tools to make dpkgs? (and you use old config)?

Reply on news.ycombinator.com