Density – Fast compression library | Hacker News Reader

Density – Fast compression library

github.com

99 pointspykello16 comments

Excelent commentary and a faster implementation by the legendary Charles Bloom: http://cbloomrants.blogspot.com/2015/03/03-25-15-density-cha...

It's funny, over the last week I've repeatedly come across things from people working at RAD Game Tools. It looks like they've got quite a good group of people working there.

They seem to be symmetric in compression and decompression. In most workloads you actually need to decompress way more often than compress.

I think this, or something like it, would be really useful with zram (Linux RAM compression), which currently uses LZO.

Why no comparsion with common contestants: gzip, bzip, dd?

Those compression algorithms are not really designed for the type of work lz4/density is. It's basically shooting fish in a barrel. You can find plenty of lz4 benchmarks comparing it to other compression algorithms.

This popped up on reddit/r/programming yesterday, so I'll just copy my comment from over there:

    How well does density [perform] when there's competition
    for the I$ and L$? LZ4 in theory is slower than a number
    of competitors (like snappy). However, LZ4 actually
    handily outperforms them when put into a "live" situations
    where it's running as a service alongside the 'real'
    program.

The size of the in-memory data-structures is the real Achille's heal of most of the contenders to LZ4 I've seen.

Interesting. I've just been a casual observer to these types of things and haven't really considered total memory used but that's a big issue on devices like mobile, or even server environments where there many be many jobs concurrently.

It's too bad that the Squash benchmark, linked in the github repo, doesn't have total peak memory usage. Do you know what the best contenders are as far as that's concerned?

The last time I researched this, to sufficient depth, was about 3 years ago. LZ4 was, hands down, the winner---nothing was even close.

Maybe because they're known to be slow, specially bzip. http://catchchallenger.first-world.info//wiki/Quick_Benchmar...

If that's the case, it would've been nice to see snappy included in the comparisons, since it seem appears targeted at the same use cases.

The README mentions Squash, where you can see the results of many more compressors. https://quixdb.github.io/squash-benchmark/

The graphs near the bottom of that page look very useful for understanding how density and other compressors compare. But, they also look dramatically different each time I refresh the page... Reporting from iOS8 Safari.

Thats because the testfile and testbed machine are chosen randomly each time the page reloads. That's mentioned in the blue note that at the top of each section."The default dataset is selected randomly."

How does it compare with Zstandard? http://fastcompression.blogspot.com.es/2015/01/zstd-stronger...

Reply on news.ycombinator.com