Intel's Storage Acceleration Library | Hacker News Reader

Intel's Storage Acceleration Library

github.com

30 pointssharva6 comments

https://01.org/intel%C2%AE-storage-acceleration-library-open... has more info

I've used ISA-L as the backend implementation for erasure codes in OpenStack Swift.

There are two versions of ISA-L. The open-source one includes some optimized erasure code libraries. The other version is free (as in beer) AFAICT but it requires signing a license agreement with Intel. This version also includes some other optimized libraries for hashing, crypto, and stuff like that.

https://github.com/openstack/liberasurecode is a plugable C library that you can use for erasure coding. It supports some simple EC algorithms itself and works with jerasure and ISA-L.

https://github.com/openstack/pyeclib is a Python wrapper for liberasurecode.

This appears to be an implementation of Reed-solomon error correction code... at the assembly level. (!! handwritten assembly? I don't see that too often nowadays)

https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_cor...

Which is basically the mathematics behind RAID-5, RAID-6, and even hard-drive / SSD sector-level error correction. Parchive is an older open-source project that implements the same concept:

https://en.wikipedia.org/wiki/Parchive

RAID-5 uses simply parity, RAID-6 uses something slightly more complicated, but they are not close in complexity (or capabilities) of Reed-Solomon codes -- although, if there was a Raid Level that had 1024 drives of which any 16 can fail, it would probably employ Reed Solomon codes.

You're right about Raid5.

My understanding is that Raid-6 is Reed-solomon however: https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6

> Several methods, including dual check data computations (parity and Reed-Solomon), orthogonal dual parity check data and diagonal parity, have been used to implement RAID Level 6

> if there was a Raid Level that had 1024 drives of which any 16 can fail, it would probably employ Reed Solomon codes.

Reed-Solomon codes are generalized for any number of drives with any number of failures.

There's a bitwise-efficiency thing with powers-of-two as well. I forgot the details, but I did the math back in college. I'll have to find my notes on this subject... but yeah, the matrix works for virtually everything.

I forgot whether or not Reed-Solomon becomes just a simple xor-parity check in the case of 1-failure, but it wouldn't surprise me if it were related.

it would be very helpful if they included a description as to what it is this library does.

Reply on news.ycombinator.com