RAD's ground breaking lossless compression product benchmarked | Hacker News Reader

RAD's ground breaking lossless compression product benchmarked

richg42.blogspot.com

80 pointsrinesh28 comments

Impressive results for RAD's suite of lossless compression products, which is apparently called "Oodle"[0] and was on HN a few months ago[1].

I can't put my finger on it, but something about the blog post bothers me. The "groundbreaking" compression suite is closed source and commercial, and even the data set used for testing in the post isn't public. Con Kolivas (ck), the author of lrzip, commented and inquired about the corpus but the blog post author says he "can't release it". I dunno, it just seems like a weird exchange to me.

[0] http://www.radgametools.com/oodlewhatsnew.htm [1] https://news.ycombinator.com/item?id=11583898

The author is a game person and primarily interested in game compression (compression at compiletime, decompression at runtime), so if he's lazy and smart, then his corpus consists of assets from the games he's worked on. That would be both conveniently available and realistic.

Of course he can't say in public that he's kept copies of any of those assets. They belong to his previous employers.

> something about the blog post bothers me

Yep, same here. Way too much excitement for something that shows no "groundbreaking" advantage over other codecs based on his own benchmark results.

Right. There's a niche where it excels, giving you lz4 decompression speed combined with zlib-esque ratios, but it accomplishes that by having significantly slower compression than either of them. It's not really breaking away from the pack in any way that I can see.

I think it's targeted at the game industry, so the slow compression doesn't matter. It just needs to decompress quickly for the user.

Go read Cbloom blog for much better information, and some implementation deets.

You don't explain what exactly is wrong with it being commercial and closed-source. That is the default way businesses make money.

It means the claims are unverifiable. The test set especially should be available.

>It means the claims are unverifiable. The test set especially should be available.

Eh..Why should anyone in the world care that a compressor can compress test data well? Except vaguely, it might serve as a motivator to investigate the use of said compressor for their own purpose. But then again, this is simply marketing, for developers.

To me, its like saying "the new compiler version is faster" on this collection of source files. Personally, I couldn't care less what the test set is, if the compiler is not faster for my own use case.

So yeah, if anyone is interested, they could probably get an evaluation copy, try it out on their own data and decide for themselves.

Of course it would be better, but I thought it was implausible that the code would be public given that the corpus is private. I guess it's possible though.

Isn't there an argument to be made for keeping the corpus private? Namely to prevent the unscrupulous from 'cheating to the test' - as games companies have a long, long history of doing.

My litmus test is checking how many "titles" authors list on their pages. The more titles the less likely I am to take their word seriously.

Pretty crude heuristic, but seems to work with very few false positives!

I feel your heuristic is simple and brilliant.

I would love to see Kraken taken for a spin by objective third parties in a fully transparent way that compared it to the available alternatives on equal footing.

Sadly, a compression benchmark that doesn't detail the compiler, hardware, corpus, or even version of the libraries used is not very compelling. If others cannot reproduce your results, the results aren't of much value beyond marketing.

Go read cbloom’s posts: http://cbloomrants.blogspot.com http://www.cbloom.com/rants.html http://www.cbloom.com/oodle_arm_report/

Stuff like the compilers used for Oodle are detailed there, and a bunch more benchmarks are published using (I believe) open corpora.

Lots more stats and graphs on Charles Bloom's blog (RAD employee working on compression) - http://cbloomrants.blogspot.co.uk/

For me the most interesting benchmark for most use cases is that from the squash library for it's plugins, because it's interactive and visually well made: https://quixdb.github.io/squash-benchmark/unstable/

And these high end benchmarks for bigger data: http://mattmahoney.net/dc/silesia.html http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/10gb.html

RAD's packer is nice for cases when compression time is not the most important aspect.

Seems like LZ4 is still unbeatable if you need as fast compression as possible. Think kernel memory page compression (to avoid swapping pages to disk) and compressing data destined to a high bandwidth sink. I'm just guessing, but probably LZ4 has also less power consumption.

If your output device can do over ~100 megabytes per second, LZ4 seems to win easily.

Optimal compression algorithm selection seems to depend on capacity and bandwidth constraints of output device.

I often use a script I hacked together called memo [1] when exploring the commandline. Today I was running a pipeline like this:

  $ memo expensive-with-lots-of-output | memo -s expensive | less

I noticed that lz4 (the default (de)compressor memo uses) was the bottleneck. ("memo -s" absorbs standard input and calculates a checksum over it and the command arguments, if that checksum is in the cache, it knows it can just output the cached value instead of running the command).

So I started looking around if there was nothing better around. From the Squish benchmarks posted elsewhere on this article [2], I noticed that zstd(1) seemed to make the right trade-offs. I required compression speed > 50MB/s, decompression speed at least as fast, and as high a compression ration as possible.

zstd(1) seems to work brilliantly and is very fast, it was easy to compile too, just running make in its directory was sufficient for me. I love those kinds of projects. I've made it the default over lz4. What gave me even more confidence in this change is that zstd is by the author of lz4.

[1]: https://github.com/aktau/dotfiles/blob/master/bin/memo (currently a version that doesn't prefer zstd(1) yet, I need to git push).

[2]: https://quixdb.github.io/squash-benchmark/unstable/

> I noticed that lz4 (the default (de)compressor memo uses) was the bottleneck.

It probably depend on how much bandwidth you got.

The RAD guys have entries in that dept too. "Selkie is 1.3X faster to decode than LZ4 and gets 12% more compression" [1] though "Selkie does not currently have a super fast encoder" [2]

[1] http://cbloomrants.blogspot.com/2016/07/introducing-oodle-me...

[2] http://cbloomrants.blogspot.nl/2016/07/oodle-selkie.html

Interesting preface to this, http://richg42.blogspot.co.uk/2016/01/zlib-in-serious-danger...

However, the compression is nothing compared to Pied Piper's algorithm.

Someone had to say it.

Any insight into which tool was used to produce these graphs?

Impressive. This reminds me of the series Silicon Valley, though.

comparison to lzturbo possible? (Unfortunately also closed source, sich).

Reply on news.ycombinator.com