Hacker News Reader

Comment by alanfranz | Hacker News Reader

Is there any system for error recovery in userspace? Something like, I’ve got directory DATA, i want to create a recovery file for all the contents in such directory. Ideally I’d like to be able to tune the size of the recovery, larger recovery = recover more errors

View on HN

What you're looking for is called Parchive. It uses Reed-Solomon coding in order to generate redundant files that can be used to recover the original. Look for PAR2 utilities (the current major version of the format). It has multiple implementations, open source and closed, across platforms and architectures.

https://en.wikipedia.org/wiki/Parchive

I wrote a gui for it, tot help deal with many files being in various states: https://github.com/brenthuisman/par2deep

Another option is to use an IBLT (Inverted Bloom Lookup Table). it is easy to implement. I recently wrote a file repair tool that uses it: https://github.com/thomasmueller/tinyStats/blob/master/src/m... (not production ready)

We used to use this when creating backups on CD-Rs and later DVDs, before hard drives became cheap...

https://en.wikipedia.org/wiki/Parchive

I used a program named Dvdisaster for the same use case. You could use it I two modes: (1) One disc with additional out of band reed-solomon ECC. Create an iso that are smaller by a certain percentage than the medium you want to burn it to. Dvdisaster will create a new ISO that uses the full capacity of the medium and fill the remaining space with parity-information that protects the whole file system and all user files. (2) Put your parity information on the next disc you burn. A parity file is crated for the first disc you burn, this is later added to the file system of the second disc you burn. Repeat indefinitely. You always have the last yet-to-be-burned parity file on your computer and can thus recover bit-errors in every disc you have burned.

We used to use this in Newsgroup days. Large files got broken into smaller chunks (maybe floppy disk sized??) in the RAR format. You could recover missing files if they expired before you get them downloaded on your dial-up modem or your particular newsgroup service wasn't reliable enough.

This isn't a turn key utility, but it's relatively easy to call from Go. It allows you to set the number of data shards and parity shards, it's pretty fast. It's at

  https://github.com/klauspost/reedsolomon

Seems like a pretty common solution, I suspect there's other libraries for popular language/platforms.

Reply on news.ycombinator.com