Show HN: TurboRLE – Turbo Run Length Encoding using SIMD | Hacker News Reader

Show HN: TurboRLE – Turbo Run Length Encoding using SIMD

github.com

23 pointspowturbo11 comments

For anyone else who didn't know:

> Run-length encoding (RLE) is a very simple form of lossless data compression in which runs of data (that is, sequences in which the same data value occurs in many consecutive data elements) are stored as a single data value and count, rather than as the original run. This is most useful on data that contains many such runs. Consider, for example, simple graphic images such as icons, line drawings, and animations. It is not useful with files that don't have many runs as it could greatly increase the file size.

> RLE may also be used to refer to an early graphics file format supported by CompuServe for compressing black and white images, but was widely supplanted by their later Graphics Interchange Format. RLE also refers to a little-used image format in Windows 3.x, with the extension rle, which is a Run Length Encoded Bitmap, used to compress the Windows 3.x startup screen.

> Typical applications of this encoding are when the source information comprises long substrings of the same character or binary digit.

https://en.wikipedia.org/wiki/Run-length_encoding

Hi powturbo! Seeing as these benchmarks are for Sandy Bridge, I thought it would be interested to see how the numbers changed for Haswell and Skylake. I couldn't immediately figure out how to replicate the benchmark, though. Could you give (or point to) some explicit directions on the steps to follow? Thanks.

hi Nathan, glad you want to benchmark TurboRLE on Haswell and Skylake. Well, it is very simple: 1- download the lkatest TurboRLE from github 2- cd TurboRLE 3- make 4- ./trle file

The enwik9bwt must be generated. The two other files can be directly downloaded. You must also download rle64 if you want to use test it.

The better option is to use TurboBench: https://github.com/powturbo/TurboBench

Where is RLE encoding used a lot nowadays? Or is this code mostly an academic exercise?

It is a great candidate anytime there is columnar storage of a database. Think of a column storage boolean value with 90% false values.

I would personally use a bit array with an associated value to indicate what 0 or 1 means.

That is a certain type of encoding. There is always overhead with this type. Columnar databases often provide lots of different encoding options (mixed in with compression) for these type of things. Products like Vertica can build the associativity for you ahead of time so you don't even have to maintain it.

Also, I believe you misunderstood how a columnar storage would work as your example would be for encoding values on a single row where I was referring to a column type.

Savestates for Rollercoaster Tycoon 1 and 2 are RLE. Makes sense as a savestate is one fixed sized/positioned datastructure with many runs of null bytes.

Doesn't fit the "a lot" usecase, but I wanted to at least give one real world example ;)

OK, a couple people posted examples. But wouldn't all of these get better compression out of an extremely low compression level gzip or something?

RCT was handwritten in Assembler, which is probably one of the reasons for using something so simple as RLE.

Ah yes of course that makes sense.

Reply on news.ycombinator.com