Hacker News Reader

Comment by whartung | Hacker News Reader

As a desktop user, I am content with APFS. The only feature from ZFS that I would like, is the corruption detection. I honestly don't know how robust the image and video formats are to bit corruption. On the one hand, potentially, "very" robust. But on the other, I would think that there are some very special bits that if toggled can potentially "ruin" the entire file. But I don't know.

However, I can say, every time I've tried ZFS on my iMac, it was simply a disaster.

Just trying to set it up on a single USB drive, or setting it up to mirror a pair. The net effect was that it CRUSHED the performance on my machine. It became unusable. We're talking "move the mouse, watch the pointer crawl behind" unusable. "Let's type at 300 baud" unusable. Interactive performance was shot.

After I remove it, all is right again.

View on HN

Back in 2011, Apple paid $500 million for Anobit, a company producing enterprise grade SSD controllers.

Since Apple was already integrating custom SSD controllers onto their A series SOCs, presumably the purchase was about Anobit patents.

> Anobit appears to be applying a lot of signal processing techniques in addition to ECC to address the issue of NAND reliability and data retention... promising significant improvements in NAND longevity and reliability. At the high end Anobit promises 50,000 p/e cycles out of consumer grade MLC NAND

https://www.anandtech.com/show/5258/apple-acquires-anobit-br...

Apple has said in the past that they are addressing improved data stability at the hardware level, presumably using those acquired patents.

> Explicitly not checksumming user data is a little more interesting. The APFS engineers I talked to cited strong ECC protection within Apple storage devices... The devices have a bit error rate that's low enough to expect no errors over the device's lifetime.

https://arstechnica.com/gadgets/2016/06/a-zfs-developers-ana...

What is the devices lifetime in apples eyes? 5 years till support is dropped? Gulp.

No, generally much longer.

Apple products go vintage after they've stopped selling something for 5 years. Obsolete after 7 years.

https://support.apple.com/en-us/102772

That's excellent if you're using Apple SSD storage, but seems unhelpful if you're running, say, a large array of HDDs connected to your Mac.

Sure, if you are managing a large amount of external data, you may want to go with a RAID array or the existing OpenZFS port, but even Windows 11 still defaults to NTFS, which offers less resilience.

> Just trying to set it up on a single USB drive

That's the fault of macOS, I also experienced 100% CPU and load off the charts and it was kernel_task jammed up by USB. Once I used a Thunderbolt enclosure it started to be sane. This experience was the same across multiple non-Apple filesystems as I was trying a bunch to see which one was the best at cross-os compatibility

Also, separately, ZFS says "don't run ZFS on USB". I didn't have problems with it, but I knew I was rolling the dice

Yeah they do say that but anecdotally my Plex server has been ZFS over USB 3 since 2020 with zero problems (using Ubuntu 20.04)

Anyway only bringing it up to reinforce that it is probably a macOS problem.

I haven't had any issues with attaching normal SATA drives with SATA-over-USB cables. Meanwhile some otherwise apparently fine USB thumb drives got abysmal results (<100kbps read IIRC).

ZFS on Linux.

> I honestly don't know how robust the image and video formats are to bit corruption.

It depends on the format. A BMP image format would limit the damage to 1 pixel, while a JPEG could propagate the damage to potentially the entire image. There is an example of a bitflip damaging a picture here:

https://arstechnica.com/information-technology/2014/01/bitro...

That single bit flip ruined about half of the image.

As for video, that depends on how far apart I frames are. Any damage from a bit flip would likely be isolated to the section of video from the bitflip until the next I-frame occurs. As for how bad it could be, it depends on how the encoding works.

> On the one hand, potentially, "very" robust.

Only in uncompressed files.

> But on the other, I would think that there are some very special bits that if toggled can potentially "ruin" the entire file. But I don't know.

The way that image compression works means that a single bit flip prior to decompression can affect a great many pixels, as shown at Ars Technica.

> However, I can say, every time I've tried ZFS on my iMac, it was simply a disaster.

Did you file an issue? I am not sure what the current status of the macOS driver’s production readiness is, but it will be difficult to see it improve if people do not report issues that they have.

A thing that can destroy most files in terms of readability is a bit flip in the header section. These could be theoretically corrected with clever guesswork and a hex editor, but in practise it is going to be hard to know where the bit flip occured, when a file just can't be read anymore.

Good catch. I had omitted that since I was not familiar with the low level details of the file formats (e.g. would a video file duplicate that data throughout it for enabling live streaming), but if the file only has one copy of important information (e.g. I-frame spacing, dimensions) and that gets corrupted, boom.

If you know that a particular JPEG block is corrupted, it could be possible to isolate the damage to just that block. You know the next block should not have a hard edge against the block above it, so you could try to find something that minimizes the vertical edge strength for the remaining blocks in the row.

> As a desktop user, I am content with APFS. The only feature from ZFS that I would like, is the corruption detection.

I think some power users would appreciate an equivalent to zfs send/recv.

Yeah, of course ZFS on macOS is a disaster, the people trying to make it work are what? Two people? One heroic dude?

But I agree with you about the most important feature that Apple failed on.

I lament Apple's failure to secure ZFS as much as the next Mac user, but I understood that ship to have sailed when it did, around a decade ago.

But as a desktop user, I almost threw up in my mouth when APFS came out with no checksums for user data. I'd assumed if they were going to pass on ZFS they would reimplement the most critical feature. But they didn't. I know I've lost data because of it, and if you have a few TB of data, I suspect you have, too.

Not lost permanently, because I had manually-checksummed backups. But lost in the sense that the copies of the data made on macOS were corrupted and different, and the OS/filesystem simply didn't notice. Oops!

Once you've used ZFS, that's unforgivable. I mean, ZFS has had some bugs, over the years (well mainly just last year lol), and a small number of users have lost data due to that. But to silently lose data by design? Fuck all the way off. It's 2025, not 1985.

So while APFS has some cool modern features (or at least two), it's an embarassment. (And all my non-ephemeral storage is now on Linux.)

> The only feature from ZFS that I would like, is the corruption detection.

I run ZFS on my main server at home (Proxmox: a Linux hypervisor based on Debian and Proxmox ships with ZFS) but...

No matter the FS, for "big" files that aren't supposed to change, I append a (partial) cryptographic checksum to the filename. For example:

20240238-familyTripBari.mp4 becomes 20240238-familyTripBari-b3-8d77e2419a36.mp4 where "-b3-" indicates the type of cryptographic hash ("b3" for Blake3 in my case for it's very fast) and 8d77e2419a36 is the first x hexdigits of the cryptographic hash.

I play the video file (or whatever file it is) after I added the checksum: I know it's good.

I do that for movies, pictures, rips of my audio CDs (although these ones are matched with a "perfect rips" online database too), etc. Basically with everything that isn't supposed to change and that I want to keep.

I then have a shell script (which I run on several machines) that uses random sampling where I pick the percentage of files that have such a cryptographic checksum in their filenames that I want to check and that verifies that each still has its checksum matching. I don't verify 100% of the files all the time. Typically I'll verify, say, 3% of my files, randomly, daily.

Does it help? Well sure yup. For whatever reason one file was corrupt on one of my system: it's not too clear why for the file had the correct size but somehow a bit had flipped. During some sync probably. And my script caught it.

The nice thing is I can copy such files on actual backups: DVDs or BluRays or cloud or whatever. The checksum is part of the filename, so I know if my file changed or not no matter the OS / backup medium / cloud or local storage / etc.

If you have "bit flip anxiety", it helps ; )

The checksum doesn’t help you fix the flipped bit nor does it tell you which bit flipped. You would have to re-create from a complete back up instead of using the efficiency of parity discs. Basically Raid 1 vs Raid 5

If OP is backing up locally onto a ZFS server like they said they were then say propagating this data to a cloud provider like Blackblaze which uses ext4 this sort of approach makes sense.

This approach is also good when you have multiple sources to restore from. It makes it easier to determine what is the new "source of truth."

Theres something to be said too for backing up onto different FS too. You don't want to be stung by a FS bug and if you do then it's good to know about it.

I wonder how hard would it be to detect which single bit was flipped? As ryao noted, in JPEGs it's immediately obvious where the image was corrupted - by visual inspection. Similar for videos, you only need to inspect the data following a single I-frame. Even for bitmap/text files, you could just scan the entire file, try flipping one bit at a time, and compare the result with the checksum.

Unlike e.g. KDFs, checksums are built to be performant, so that verifying one is a relatively fast operation. The Blake family is about 8 cycles per byte[1], I guess a modern CPU could do [napkin math] some 500-1000 MB per second? Perhaps I'm off by an order of magnitude or two, but if the file in question is precious enough, maybe that's worth a shot?

[1]: https://en.wikipedia.org/wiki/BLAKE_(hash_function)

What if you had two bit flips?

That would be a terrible coincidence :,) however improbable, still possible.

Even proper high-end server-grade ECC ram guarantees "only" SECDED (single error correction, double error detection).

There are tools that give you hashing + parity, e.g. SnapRAID: https://www.snapraid.it/

+1 For Snapraid.

If you already have some data on ext4 disk(s) and don't want to deal with the the issues of using ZFS/BTRFS then it's a no brainer. Dynamically resizing the "array" is super simple and it works really well with MergerFS.

Reply on news.ycombinator.com