While we're talking about compression algorithms, here is a nice little gem called BMZ:
http://bitbucket.org/mattsta/bmz/src
It's is a fast compression scheme implemented using BMDiff and a Google Zippy clone (based off LZO).
http://bitbucket.org/mattsta/bmz/src
It's is a fast compression scheme implemented using BMDiff and a Google Zippy clone (based off LZO).
The results of my unscientific test (compression only):
Compressor Size Ratio Time
gzip -1 23MB 88% 1.18s
gzip -2 23MB 87% 1.38s
bzip2 23MB 87% 5.57s
xz -1 23MB 87% 5.35s
xz -9 11MB 43% 10.58s
bmz 13MB 45% 0.95sBMZ = bmpack + lzo by default and can be combined with lzma if necessary. It's not really a BMDiff and Zippy clone, as I've never had a chance to see Google's implementation. It's based on the original Bentley & McIlroy paper: "Data Compression Using Long Common Strings", 1999. Even the two pass idea is from that paper. It was really a wacky experimental implementation (with a lot of room for improvement) to satisfy my curiosity. I'm a little surprised that the 0.1 version has been stable for quite a few people compressing TBs of data through it.