Hacker News Reader

Comment by wmf | Hacker News Reader

BTW, bz2 is kinda over. Check out xz and the parallel version pxz.

I've also got my version of parallel xz: https://github.com/vasi/pixz

It doesn't require use of large temporary files like pxz, and the xz files it produces can also be decompressed in parallel.

Looks worth trying out --

Can I suggest adding installation packages for: brew ubuntu (at least through launchpad)

vasi, thanks very much for writing pixz, I've had a great experience working with it and it sped up my code very significantly.

The only thing I wish for is more documentation and inclusion in xz-utils (which I know is not up to you, but I'm hopeful).

Glad to hear it's been of use :)

The author of xz is working on a parallel compression implementation of his own, which will hopefully be in future versions of xz.

That looks very promising. As others have pointed out here, pixz would be much more accessible if one could simply install it via "brew install pixz" or "aptitude install pixz".

I welcome anybody who'd like to package it! I think it's already available in Arch Linux.

I'm no expert in compression, so its no surprise that I've never heard of xz. What are it's strengths compared to gzip, or b2z? Efficiency? Convention?

You probably heard of 7Zip and LZMA. xz uses the LZMA2 algorithm to achieve better compression ratios.

You've probably heard of LZMA, xz is merely an implementation of this (7zip and rar also use LZMA). The most notable differences to bzip2 are significantly longer compression time and rather moderate decompression speed increases. This makes xz quite good for packaging data for distribution, although it takes a lot longer to compress, it decompresses a lot faster (unlike bzip2) and saves a significant amount of space.

I disagree with your assessment. I'm aware of xz and use it when file size is paramount -- which, by the way, is almost never. Most of the time, I'm willing to give up a few megabytes in order to save time, and in my tests pbzip2 crushed xz by a factor of ten when it comes to speed.

Why not use parallelized versions of xz such as pxz or pixz? Because pre-built pxz/pixz packages are nearly non-existent. When that changes, I'll consider switching formats.

Remember that compressors, including xz, support multiple compression levels. The default level for xz is 6, which is perhaps too far on the small-but-slow side. Levels 2 and lower tend to give similar compression levels to bzip2, and are considerably faster.

Also, note that decompressing bzip2 is very slow, xz usually beats it by a factor of two or more.

I agree that the default level (6) for xz probably errs too much on favoring file size over speed. My tests with compression levels 1-2 do indeed show modestly improved size and speed performance relative to single-threaded bzip2.

The fact remains, however, that I can't seem to find a simple way to install a parallelized version of xz. Perhaps I'll post an issue in the Github issue tracker for pixz and see if we can't resolve that. :)

Recently when compressing some SQL backups, I found xz to be very slow when compared to bz2 (at least with default compression settings).

Edit: And in this case, bz2 even gave better compression ratios if I recall correctly.

If speed matters more than compression then consider lzop(1).

Reply on news.ycombinator.com