Hacker News Reader

Comment by jaytaylor | Hacker News Reader

Good riddance; for text files, gzip is mediocre compression and fast, lzma (xz) is high compression and slow. Bzip2 is medium-high compression and _really_ slow [0].

[0] http://tukaani.org/lzma/benchmarks.html

View on HN

Your link shows bzip2 having a more balanced compression time than lzma that is often 3-4 times slower. So for a tarball xz sounds indeed better as you compress once and uncompress many times. For compressing HTTP or things of the sort bzip2 might be better, but maybe for those cases gzip is already best.

You are right, I was speaking to the compression ratios and decompression times. For cases where something is compressed once and decompressed many times I don't care much how long it takes to do the initial compression (within reason, of course).

Here is another with lz4, lzo as well as lzma1, although they're not very relevant for the tarball test. Gzip -9 is on par with bzip2 -1 while almost twice as fast. And uses practically no memory(!).

http://pokecraft.first-world.info/wiki/Quick_Benchmark:_Gzip...

-1 is an edge-case: if you're memory constrained, bzip2 isn't a good fit. Otherwise, you pay the extremely low percentage increase in CPU time to use -9 where bz2 will massively outperform gzip on most workloads.

Note that LZMA decompressed at 20MB/s in that benchmark, while bzip2 managed about 6.

LZMA tends to be faster and more efficient than bzip2 in all respects.

pedrocr said that LZMA decompression is faster, but that sometimes LZMA compression is slower.

I don't know how well lzma parallelises, but bzip2 is plenty fast for me with a multi-core version like pbzip2, as measured in wall-time on an otherwise idle system. Different considerations in different environments and all that.

Unfortunately there's no multithreaded option for the standard xz uncompression tools (xzcat, unxz). However I wrote (as a quick hack) a parallel xzcat:

http://git.annexia.org/?p=pxzcat.git;a=summary

I also wrote a random access program for xz files:

https://github.com/libguestfs/nbdkit

(in the plugins/xz subdirectory)

The version of the tools that I have installed (5.1.2alpha in Fedora) do support a -T option for parallel compression which I use regularly.

However, it is true that that same option does not seem to work everywhere. At least, it didn't seem to work a few weeks ago in a different system (Ubuntu). The option is accepted, but does not seem to have any effect.

lzma parallelizes very well! My implementation of parallel xz ( https://github.com/vasi/pixz ) works quite well. There are others as well, including pxz and the alpha version of the standard xz. All these tools produce compressed files that conform to the xz format and can be decompressed by any xz utility.

I'll throw out a vote for lbzip2. It threads strongly, and doesn't cannibalize your memory like threaded xz. (You can still use gzip where every CPU cycle counts)

Reply on news.ycombinator.com