Hacker News Reader

Comment by spicyj | Hacker News Reader

Can someone explain to me why this Wikipedia article is at the top of the front page?

Most hackers don't know about XZ yet, but they should. I still prefer not to see Wikipedia articles on HN since they lack context. A "7 reasons you should ditch bzip2 and use XZ" blog post would probably be better.

Oh god please not more "X reasons for Y"

Give me Z reasons why you don't like "X reasons for Y" posts.

9 Reasons Why You Should Hate "X Reasons for Y" posts:

1. They're overdone

2. They're uncreative

3. They're unoriginal

4. They're a crutch

5. They're assembly-line writing

6. They're used by tabloids to appeal to supermarket zombies

7. They've helped turn respectable mags into tabloids (or are a symptom of it, not sure which came first)

8. They're psychologically manipulative (I don't know how, I just know it)

9. They work

You forgot to put these across 11 individual comments so your 8pts would have been multiplied to 88!

I do not mind lists, but I cannot stand lists that are seperated to pages for inflated page counts!

He was going to put it across 11 comments, but as the Wikipedia article clearly points out, xz compression conventionally bundles a single file.

The HN Guidelines explicitly frown upon "X reasons..." titles because they're almost always linkbait.

If the original title begins with a number or number + gratuitous adjective, we'd appreciate it if you'd crop it. E.g. translate "10 Ways To Do X" to "How To Do X," and "14 Amazing Ys" to "Ys." Exception: when the number is meaningful, e.g. "The 5 Platonic Solids."

http://ycombinator.com/newsguidelines.html

How about a "Y is the new X" or "X is dead, long live Y"?

I'd prefer it to a two paragraph Wikipedia article.

"X reasons for Z": there, I fixed it for you.

Agree on the wikipedia articles with no context point. Especially since the article in question is short with not very much real information on why it matters.

LZMA2 has a better compression ratio than bzip2, according to this article http://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Mark... . Seems like it would be a good idea to switch away from bzip2/gzip to zx

The main reason to continue to use gzip is speed: it's an order of magnitude faster than bzip2 for compression / decompression, for a relatively small compression ratio hit. It's fast enough to us inline in web servers, SSH sessions, etc, and if you're just moving data around a single time, often the amount of time for compress / transfer / decompress is much faster than with bzip2.

I've found that on many types of data, xz -2 is twice as fast and compresses better than gzip -9. Of course gzip -1 kicks the pants off xz but there's absolutely no reason to use gzip at its higher levels, nor to use bzip2 at all. And if you want fast, LZO does it better -- the lzop tool provides a typical unix compress interface.

Using 50MiB from a tarball I had lying around, best of 3:

     prog    time(s)   size(%)
    gzip  -1  1.703    38.7
    gzip  -9 14.670    35.1    (worse than xz -1)
    bzip2 -1  7.714    34.0
    bzip2 -9  8.035    31.4    (worse than xz -1)
    xz    -1  6.278    31.3
    xz    -9 42.445    22.7    (best compression)
    lzop  -1  0.670    46.1    (fastest)
    lzop  -9 18.144    38.0

If anyone is still reading this thread, I made a chart overnight using a better corpus:

http://partiallystapled.com/~gxti/misc/2010/06/25-shootout.p...

http://partiallystapled.com/~gxti/misc/2010/06/25-linux-2.6....

Out of curiousity, can you run this tarball through paq (say, paq8hp12)?

http://en.wikipedia.org/wiki/PAQ

It segfaulted at the end, after taking 7m28s. I don't know how big the result was because it got truncated to 35 bytes.

Ugh. Can you try with a lighter compression level? There is a command line switch for that, and it should also show the memory requirements for each compression level.

I'm insisting because PAQ is a really incredible collection of compressors. Far more intelligence in it than in the LZ* bunch.

Yes, the new compression algorithms (BWT and LZ77-based) are slower and require a lot more memory than DEFLATE (*zip) in most cases.

This guy did a not so rigorous analysis but it is mostly OK: http://changelog.complete.org/archives/931-how-to-think-abou...

A little bird told me there's a better algorithm in speed/compression rate coming soon ;)

Another feature of gzip which may be your life-saver is that it keeps a record of the original size so you can know it without decompression.

xz format seems to have this feature, but: $ xz --list xz: --list is not implemented yet.

More importantly, xz decompresses quickly. Essentially, you get the compression ratio of bzip2 at the decompression speed of gzip (and, unfortunately, the compression speed of bzip2).

EDIT: also, like gzip but unlike bzip2, you can stream data through xz (at an insignificant penalty in compression ratio).

LZMA is much, much slower. It is simply a matter of the rate at which you generate data and the compression ratio and the rate at which you add more storage/move the files off to tape. Can you "keep up" essentially.

Right now, CPUs are fast enough that LZMA is a realistic prospect. It boils down to what the extra CPUs will cost vs the cost of the storage saved (incl. the "cost" of space in the datacentre, the administrative overhead of more hardware, etc).

I wonder if it would be possible to have page templates delivered separately from their dynamic content? The templating of the dynamic data would happen on the browser-side machine. This way, we could use really expensive compression algorithms with high CPU cost for compression, but then cache the compression result.

This could be achieved by using Javascript and fetching data as JSON, but it seems to be something that would be very useful and beneficial if it were standardized.

There is the SDCH spec: Shared Dictionary Compression over HTTP. Google Chrome uses it, and the Google Toolbar adds support to IE.

The server supplies a pre-created dictionary (template) which gets cached, and further requests get a diff to that dictionary as a response.

I've only found Google search using it on the server side, and the browser penetration is fairly low, but it seems promising. It makes requests very fast.

I've also noticed that Google search serves image thumbnails as data:// URLs directly in the original response. They are going a long way to ensuring each page loads entirely in one request.

There is client-side XSLT already. The problem is that transformed template is usually same size or smaller than your source data.

Really expensive algorithms aren't that much better to offset cost of sending template logic to the client.

You could probably save some processing time by integrating compression directly into server-side templating - take advantage of the fact that some parts never change and keep them pre-compressed or at least cache some statistics about them to aid compression.

In retrospect, I probably should have submitted the XZ Utils project page: http://tukaani.org/xz/ .

However, that page does not give the same picture of widespread adoption that the Wikipedia article does. That use was what interested me: I'm surprised that it only came to my attention when I was trying to recall the name of the 7z command line utility p7zip.

I upvoted it because, on top of being interesting, it fights the passively accepted wisdom that "newsworthy" is synonymous with "worthy".

I'm guessing because people voted it up. Additionally, it's not a peak time right now.

When is peak time?

It's apparently something new that has been floating around. Some hackers are probably curious about the "new thing" that popped up lately.

Software that was released 9 months ago isn't particularly new.

First, I think that is why it's in quotes.

Second, I am reading Of Mice or Men for the first time this summer. It is new to me!

Reply on news.ycombinator.com