Hacker News Reader

Comment by elfchief | Hacker News Reader

Is it just me or is it absolutely insane that zfs and btrfs are the only common filesystems out there that do data checksumming? I don't want or need the extra complexity of either of them in a lot of cases, but I'd sure as hell like to know if my data is corrupt...

View on HN

Yeah it should be standard. Equally crazy is that APFS is brand new and didn't add it. I belive the justification was that the hardware does it on flash memory for their devices, but ZFS really exposed how unreliable hardware is.

I mean Apple has their own flash controllers, so they’re really the only company that owns the FS and the flash controller. So really they can depend on it.

What about external hard disks with terrible USB controllers? That's where data checksumming would be even more important and where Apple doesn't produce the hardware.

Maybe APFS was mainly developed for iOS?

Apple doesn’t recommend using APFS for external media.

Really? Can you remember where you've seen this non-recommendation?

Apple isn't really embracing APFS for external drives as I see it. Time Machine backups would be one of the main uses for external drives, but…

> Time Machine can’t back up to an APFS-formatted disk. If you select a back up disk formatted as APFS, Time Machine offers to reformat it as HFS+.[1]

[1] https://support.apple.com/guide/mac-help/types-of-disks-you-...

This has more to do with the implementation details of TimeMachine which relies on features of HFS+ not available in APFS. It's not Apple telling you not to use APFS on external drives.

I can't recall if I ever saw it, but by default it will use HFS+ for external drives.

Every time I unplug an external USB prematurely, the next time I plug it into my mac, it does a full filesystem check of it which takes a few mins.

There have been at least two occasions in the past three years that I remember where Apple had to recall or issue firmware updates for SSDs due to corruption issues.

It’s a big IF to depend solely on flash based error checking. I think Apple likely figured the cost of implementation and system overhead wasn’t worth it for Macs and iOS devices.

I’d be very surprised if they use APFS or a non checksumning file system on their iCloud server farm though.

There is also windows ReFS and Dragonfly's HAMMER2.

BcacheFS has it & is under active development: https://bcachefs.org/

A lot of applications and databases do this themselves.

Part of the problem is that the app might know what the proper checksum is and it would be nice to be able to write a document ahead of time and tell the FS that you already know the checksum.

Won't save a massive amount of CPU but it would be worth it if it was easy to implement.

There is however no point in having data checks if there are no way of recovering, so running ZFS only make sense if you have multiple disks. And don't forget to scrub your pool!

This is like saying "there's no point in ECC if it can't correct every error" ... which simply isn't true. It's still far better to know that corruption exists -- so you can know something is invalid and potentially take action -- than to have corruption silently hanging around doing corrupted data things.

That is not a fair comparison as ECC is Error Correcting, while just check-summing is not. If you however have a mirror or raid-Z the file system will be able to correct.

Knowing that your data is corrupt is always better than not knowing.

Whether or not you have a RAID you should still have backups, and you don't want to replace a good backup by a corrupt one.

OK, so you know your HDD is corrupted, and you now have to take the site down to recover from backup, which is also old, so there will be data missing. Compared to if the FS can just fix itself. Sometimes not knowing can be a bliss though. :P

You can set the number of copies to keep at filesystem level. This is in addition to any redundancy provided at pool level (if any). It won't save you from a failed disk, but it will be able to transparently correct any checksum error.

There definitely is a point. ZFS can tell you which files are corrupted and you can restore them from a backup.

Reply on news.ycombinator.com