Hacker News Reader

Comment by earthrise | Hacker News Reader

I actually didn't compare gzip and bzip2 before uploading, and I probably should have. I did use `gzip -9`. I'm compressing it with bzip2 -9 now to see if it gets any smaller. I'll post a reply here when I know.

View on HN

Use xz. Debian, Gnome, Archlinux, Gentoo, Fedora have all switched to xz, and for good reason. I expect xz will offer a 20-30% reduction in file size. That will add up quick.

Thanks, i'll run xz too!

My download has slowed to a crawl around 90% completion. I am doing 3k a second. When I read that your list was not sorted case sensitively I wondered what difference that may make.

I did a little experiment with the american-wordlist-insane wordlist and sorting in the interim. I used msort which does a case sensitive sort[1] and sort which does not. Here are the results:

    6830085 insane.msort
    1461684 insane.sort.xz
    1376548 insane.msort.xz

It is not a big difference but it is still a little better. For future releases you may want to do a case sensitive sort.

[1] Zebra comes before aardvark

bzip2 saved about 200mb:

gzip: 4293 bzip2: 4097

xz is still running

For anyone interested, p7zip set for -mx9 got the file down to 2.72GB on my Mac.

I don't know about p7zip but the xz man page cautions against blindly using -9.

Reply on news.ycombinator.com