Hacker News Reader

Comment by keenerd | Hacker News Reader

No forward error correction either. It can't be considered "final" without some form of bitrot protection.

And no, storing it on a ZFS or BTRFS volume with error recovery enabled does not count. (They don't use FEC, they use 1960's triplicate storage. Hugely wasteful of space, does nothing to protect against transmission errors and can still be corrupted by two of the exact wrong bits being damaged.)

Storing it on a media that does use FEC also does not count. I want a per-file tunable FEC knob, not one vendor-determined setting. And as history has shown, it needs to be done through FOSS code and not trade secret firmware.

View on HN

The FEC is implemented in the hard drive firmware. You're going to read back a full correct block or the whole block will fail. You're not going to read back a block with a single bit error, so it is insanity to protect against that at the FS level.

Also read up on RAID levels. RAID5/6 or ZRAID7 uses error correction not duplication.

What about over the net? One of the listed strengths is its streaming capabilities, which implies sending it over the net. Error correction would be important there as well.

So you need something other than IP checksums, and whatever error correction is also operating in the layers below it?

I've downloaded and uploaded many files which have arrived incorrectly over the internet, so yes, if you truly want to prevent bitrot, you do want to add error correction.

This is a transport problem, however. Why not calculate the error correction codes on the fly? Why wast disk blocks?

Because you want to guarantee a certain error margin, regardless of the error detection & retransmit/correct capability of whatever networks/file systems/media it traverses in the interval till you want to read it.

Is there any FS that implements FEC?

Not that I'm aware of. The device layer usually does it already and it's supposed to be sufficient.

> it's supposed to be sufficient.

Supposed to. If you do any backups to DVD/BD and still want to get your data back in 5 or 10 years, though, you'd be well advised to do some sort of FEC - burn multiple copies of each disc, generate a bunch of PAR2, whatever.

(You might want to do that for backups on hard drives too. Yeah, maybe the hard drive firmware is supposedly taking care of any errors below the block level and you're not too worried about bitflips, but that just means you'll lose entire blocks and files when you lose something.)

If you count the CD and DVD spec books as file systems, yes.

BTRFS has bitrot-protection

Reply on news.ycombinator.com