Hacker News Reader

Comment by matthavener | Hacker News Reader

LZMA2 has a better compression ratio than bzip2, according to this article http://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Mark... . Seems like it would be a good idea to switch away from bzip2/gzip to zx

View on HN

The main reason to continue to use gzip is speed: it's an order of magnitude faster than bzip2 for compression / decompression, for a relatively small compression ratio hit. It's fast enough to us inline in web servers, SSH sessions, etc, and if you're just moving data around a single time, often the amount of time for compress / transfer / decompress is much faster than with bzip2.

Yes, the new compression algorithms (BWT and LZ77-based) are slower and require a lot more memory than DEFLATE (*zip) in most cases.

This guy did a not so rigorous analysis but it is mostly OK: http://changelog.complete.org/archives/931-how-to-think-abou...

A little bird told me there's a better algorithm in speed/compression rate coming soon ;)

I've found that on many types of data, xz -2 is twice as fast and compresses better than gzip -9. Of course gzip -1 kicks the pants off xz but there's absolutely no reason to use gzip at its higher levels, nor to use bzip2 at all. And if you want fast, LZO does it better -- the lzop tool provides a typical unix compress interface.

Using 50MiB from a tarball I had lying around, best of 3:

     prog    time(s)   size(%)
    gzip  -1  1.703    38.7
    gzip  -9 14.670    35.1    (worse than xz -1)
    bzip2 -1  7.714    34.0
    bzip2 -9  8.035    31.4    (worse than xz -1)
    xz    -1  6.278    31.3
    xz    -9 42.445    22.7    (best compression)
    lzop  -1  0.670    46.1    (fastest)
    lzop  -9 18.144    38.0

Out of curiousity, can you run this tarball through paq (say, paq8hp12)?

http://en.wikipedia.org/wiki/PAQ

It segfaulted at the end, after taking 7m28s. I don't know how big the result was because it got truncated to 35 bytes.

Ugh. Can you try with a lighter compression level? There is a command line switch for that, and it should also show the memory requirements for each compression level.

I'm insisting because PAQ is a really incredible collection of compressors. Far more intelligence in it than in the LZ* bunch.

If anyone is still reading this thread, I made a chart overnight using a better corpus:

http://partiallystapled.com/~gxti/misc/2010/06/25-shootout.p...

http://partiallystapled.com/~gxti/misc/2010/06/25-linux-2.6....

Another feature of gzip which may be your life-saver is that it keeps a record of the original size so you can know it without decompression.

xz format seems to have this feature, but: $ xz --list xz: --list is not implemented yet.

More importantly, xz decompresses quickly. Essentially, you get the compression ratio of bzip2 at the decompression speed of gzip (and, unfortunately, the compression speed of bzip2).

EDIT: also, like gzip but unlike bzip2, you can stream data through xz (at an insignificant penalty in compression ratio).

LZMA is much, much slower. It is simply a matter of the rate at which you generate data and the compression ratio and the rate at which you add more storage/move the files off to tape. Can you "keep up" essentially.

Right now, CPUs are fast enough that LZMA is a realistic prospect. It boils down to what the extra CPUs will cost vs the cost of the storage saved (incl. the "cost" of space in the datacentre, the administrative overhead of more hardware, etc).

I wonder if it would be possible to have page templates delivered separately from their dynamic content? The templating of the dynamic data would happen on the browser-side machine. This way, we could use really expensive compression algorithms with high CPU cost for compression, but then cache the compression result.

This could be achieved by using Javascript and fetching data as JSON, but it seems to be something that would be very useful and beneficial if it were standardized.

There is client-side XSLT already. The problem is that transformed template is usually same size or smaller than your source data.

Really expensive algorithms aren't that much better to offset cost of sending template logic to the client.

You could probably save some processing time by integrating compression directly into server-side templating - take advantage of the fact that some parts never change and keep them pre-compressed or at least cache some statistics about them to aid compression.

There is the SDCH spec: Shared Dictionary Compression over HTTP. Google Chrome uses it, and the Google Toolbar adds support to IE.

The server supplies a pre-created dictionary (template) which gets cached, and further requests get a diff to that dictionary as a response.

I've only found Google search using it on the server side, and the browser penetration is fairly low, but it seems promising. It makes requests very fast.

I've also noticed that Google search serves image thumbnails as data:// URLs directly in the original response. They are going a long way to ensuring each page loads entirely in one request.

Reply on news.ycombinator.com