Hacker News Reader

Comment by water9 | Hacker News Reader

The checksum doesn’t help you fix the flipped bit nor does it tell you which bit flipped. You would have to re-create from a complete back up instead of using the efficiency of parity discs. Basically Raid 1 vs Raid 5

View on HN

There are tools that give you hashing + parity, e.g. SnapRAID: https://www.snapraid.it/

+1 For Snapraid.

If you already have some data on ext4 disk(s) and don't want to deal with the the issues of using ZFS/BTRFS then it's a no brainer. Dynamically resizing the "array" is super simple and it works really well with MergerFS.

If OP is backing up locally onto a ZFS server like they said they were then say propagating this data to a cloud provider like Blackblaze which uses ext4 this sort of approach makes sense.

This approach is also good when you have multiple sources to restore from. It makes it easier to determine what is the new "source of truth."

Theres something to be said too for backing up onto different FS too. You don't want to be stung by a FS bug and if you do then it's good to know about it.

I wonder how hard would it be to detect which single bit was flipped? As ryao noted, in JPEGs it's immediately obvious where the image was corrupted - by visual inspection. Similar for videos, you only need to inspect the data following a single I-frame. Even for bitmap/text files, you could just scan the entire file, try flipping one bit at a time, and compare the result with the checksum.

Unlike e.g. KDFs, checksums are built to be performant, so that verifying one is a relatively fast operation. The Blake family is about 8 cycles per byte[1], I guess a modern CPU could do [napkin math] some 500-1000 MB per second? Perhaps I'm off by an order of magnitude or two, but if the file in question is precious enough, maybe that's worth a shot?

[1]: https://en.wikipedia.org/wiki/BLAKE_(hash_function)

What if you had two bit flips?

That would be a terrible coincidence :,) however improbable, still possible.

Even proper high-end server-grade ECC ram guarantees "only" SECDED (single error correction, double error detection).

Reply on news.ycombinator.com