Shielding your files with Reed-Solomon codes (2008) | Hacker News Reader

Shielding your files with Reed-Solomon codes (2008)

users.softlab.ntua.gr

27 pointsTomte15 comments

> the way storage media devices have been going...

My experience is that drives have been getting substantially better in this regard. If a drive is failing, it will usually collapse into read only mode and let you know that it is failing, giving you ample time to recover data.

That said, I do think backups are a good idea, and that Reed-Solomon codes are a good way to protect your data. I also think that managing bitrot is really the responsibility of the drive itself and maybe of the filesystem. Zfs and btrfs both offer bitrot protection.

And even solving bitrot, risks like fires, earthquakes, or war (for less fortunate countries) mean that data should be backed up in multiple physical locations anyway.

These are problems I've been attempting to address at https://sia.tech for the past two years. It's still in beta, but a full platform should be ready by March.

There are also distributed file systems (MooseFS, Gluster, Ceph) which can duplicate data across physical locations, protect it with checksums, scrub in a distributed manner and auto-heal, all of that transparently.

How easy are those to setup and how reliable are they?

About hard as setting up any UNIX daemon on a few computers. Reliability is a complex topic, depends on what you need -- for an anecdotal evidence I maintain a microscopic (10-20TB/4-6 desktop hosts) Moose deployment for a distributed home dir use-case over 5 years without any data loss despite two full computer losses, one disk malfunction and numerous network and power outages (it never detected a bit-rot event though). There are many serious success and horror stories one Google query away.

> If a drive is failing, it will usually collapse into read only mode and let you know that it is failing, giving you ample time to recover data.

Do you have any real world references of this kind of fail behavior occurring? I only have experience with mechanical HDD failure (RAID drives gone bad -> degraded until rebuild; non-RAID drives getting massive bad sectors; non-RAID drives click -> dead). I've read about drives supposedly being able to enter a "nice" readonly mode, but that was a couple years or more ago, about SSDs. More recently, I've only read about drives either entering readonly mode until power cycle, then completely inaccessible; or simply lockup and completely inaccessible.

The idea is excellent (I've lost files to bitrot before), but what is the advantage here over using "Parchives" [1]? There is widespread cross-platform software available for "par2" format.

[1] https://en.wikipedia.org/wiki/Parchive

I tried to use it with my backup folder and the Windows tool I used (not sure which) dies from too many files.

I gave up after that.

Par (at least version 1) uses Reed-Solomon codes. It says right there in the article you linked.

Par1 used Reed-Solomon error correction to create new recovery files.

Right, par2 does as well [1]. My question is: what advantage does the freeze/melt thing have?

It looks to me that freeze/melt produces a larger file that contains the recovery blocks, while par2 stores the recovery blocks in a second file (or files).

If the recovery bits are stored at the end of the freeze/melt file then you can still use your frozen file as long as your program doesn't barf on extra data at the end, while the par2 approach doesn't have that limitation.

[1] http://parchive.sourceforge.net/docs/specifications/parity-v...

The freeze/melt tool is meant for solving one particular problem: sector errors on block devices such as hard drives, floppies, and CDs. Par, on the other hand, was designed for posting to Usenet binary groups where files are split into many small chunks and the loss of some chunks was so common as to be expected.

Multithreaded (Intel TBB) version of par2cmdline (archive.org link):

http://web.archive.org/web/20150607135928/http://chuchusoft....

(Apologies for not directly answering the question.)

> Storage media are of course block devices, that work or fail on 512-byte sector boundaries

It's been my experience that the entire disk typically stops working at once. "Bad sectors" are a 90's problem.

I used to do backups to optical media with freeze.sh and melt.sh. It worked well; I never had a problem restoring files.

Said optimal media already use RS encoding internally and theoretically there shouldn't be problems with these either. Even if you pile several layers of RS on top of each other you can still have bad luck.

Better to have several backups. :-)

We're unlikely to see another mass-adopted optical media format be released, but if we did, being able to set the amount of error correction myself would be at the top of my wish list. The option to fill any empty space on the disc with EC would make me sleep better at night. Yes, there are utilities which can do this now, but it's not ubiquitous in a way which guarantees I'll have access to that utility in 20y+.

Reply on news.ycombinator.com