Comment by ChuckMcM | Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

Parent Full thread

Hey Brian! For the record I love that you are transparent about your assumptions, it is really helpful. I have been in very very similar debates, both at NetApp and at Google of all places. Peter Corbett, the guy who invented the dual parity scheme that NetApp uses, wrote a similar analysis as well for Fast '04 [1].

As someone who likes to geek out on failure proof systems and perfectly secure systems, neither of which are attainable but can be asymptotically approached, I think you are seeing the "there is always one level deeper" kinds of discussions. Personally I think of them as endorsements because if the exceptions get too extreme (say 'what if an asteroid hits?') then you know you've got all the bases covered.

Its only a problem if the person analyzing the analysis finds something that you really did not even consider. Then it opens up an opportunity to look at the problem a whole new way.

[1] https://www.usenix.org/legacy/publications/library/proceedin...

Good link! I've both forwarded it on, and will study it when I have some time later.

Hi Brian, thanks for open-sourcing Backblaze's JavaReedSolomon, which is really well-written. A few months ago I ran into an issue with Reed Solomon coding throughput not saturating the write throughput of 16 drives, and wrote a new Reed Solomon module based on Cauchy matrices: https://github.com/ronomon/reed-solomon

The Cauchy matrices remove the need for a table lookup to do the Galois multiply, replacing it with pure XOR. Together with other optimizations, this gives nearly 3x-5x more coding throughput for the same (17,3) parameters, assuming you're still using your open-sourced JavaReedSolomon in production. I don't know if Reed Solomon coding throughput is a factor in your rebuild times?

There is also Intel's ISA-L: https://github.com/01org/isa-l

It contains optimized Galois Field multiplication, resulting in Reed Solomon (both Vandermonde as well as Cauchy) at multiple GB/s on a modern x86 CPU.

Still, I doubt that the Reed Solomon coding speed is the limiting factor in their rebuild time. There is a mention of a 6-day duration, so even with a very slow Reed Solomon implementation ( ~ 100 MB/s) that should not be a bottleneck for a 10 TB drive rebuild (assuming a distributed rebuild approach, not a traditional RAID style rebuild).

Reply on news.ycombinator.com