Are there any uses cases for Reed-Solomon at the application level?
My impression was always that error checking was implemented at the hardware level, or else at the OS/driver level.
But just curious if there are some applications I'm missing.
My impression was always that error checking was implemented at the hardware level, or else at the OS/driver level.
But just curious if there are some applications I'm missing.
Hadoop has an RS implementation inside the filesystem (called "erasure coding"), instead of storing 3 copies of the same data, it can actually instead store ~1.5 copies as (6+3) or (10+4).
Previously, I've run into this tech in satellite internet gateways, but distributed filesystems is where I've gone through the math & probabilities of failure properly.
I work on perf & the extra network hops (with 3 replicas, you read 100% of data local, when you stripe it that doesn't work) and math for the error correction are hot spots when you are trying to keep all cores busy.
The professor later spun up a startup that has since been acquired by Dropbox, but I'm unaware what kind of product they're currently working on.
I believe parchive version 1 was simple XOR parity.
Also, the parity matrix for v1 would be sometimes singular (non-invertible), which v2 tried to fix but it didn't quite work.
You might also be building a distributed system, where blocks are spread across multiple disks. In that case, an "error" you're trying to correct might be the loss of a disk or server in the distributed system; there is no one single disk or OS responsible for all the relevant blocks in that case.
But, both of these are assuming you're building some kind of data-store, which is not a typical user application. Writing robust data-stores is hard for many reasons, error correcting is just one of them.