Hacker News Reader

Comment by d33 | Hacker News Reader

I've recently been wondering: could you re-compress gzip to a better compression format, while keeping all instructions that would let you recover a byte-exact copy of the original file? I often work with huge gzip files and they're a pain to work with, because decompression is slow even with zlib-ng.

View on HN

precomp/antix/... are tools that can bruteforce the original gzip parameters and let you recreate the byte-identical gzip archive.

The output is something like {precomp header}{gzip parameters}{original uncompressed data} which you can then feed to a stronger compressor.

A major use case is if you have a lot of individually gzipped archives with similar internal content, you can precomp them and then use long-range solid compression over all your archives together for massive space savings.

> A major use case is if you have a lot of individually gzipped archives with similar internal content, you can precomp them and then use long-range solid compression over all your archives together for massive space savings.

Or even a single gzipped archive with similar pieces of content that are more than 32KB apart.

That's called `pristine-gz`, part of the `pristine-tar` project.

Thank you! It seems to be what I'm looking for.

I may be misunderstanding the question but that should be just decompressing gzip & compressing with something better like zstd (and saving the gzip options to compress it back), however it won't avoid compressing and decompressing gzip.

Reply on news.ycombinator.com