Hacker News Reader

Comment by VWWHFSfQ | Hacker News Reader

> older applications where gzip is unavoidable despite its obsolescence

Is gzip actually obsolete or are there just newer alternatives? gzip is still everywhere

The zlib implementation is not optimal in any way that I am aware of. You can get better compression with the same compute time, and you can get the same compression with less time, in every application I have seen, by using some other thing. I usually want to look at zstd, lz4, and brotli in addition to zlib.

That said, the igzip re-implementation is really good. If you can use it, igzip moves gzip closer to the optimal frontier.

There's also the various zlib forks by cloudflare et al, if you want to go down that road.

But yes, lz4 and zstd would be what I'd recommend people use.

The code quality of some of the cloudflare changes is questionable. Optimizations aren't cleanly separated by commits, some of the licensing is dicey, etc.

zlib-ng tried to merge as much as they could from the cloudflare fork without getting into licensing issues.

Oh, sure, it's all exciting in that respect from some of those, but I was more pointing to the collection of forks as a whole as a thing to evaluate if you have to look at zlib.

I would just go for lz4 and zstd, really, at this point, but if for some reason gzip is necessary or fits your use case better, ...

(Though I'd really like to hear what that is.)

The only place gzip wins is in compatibility. Everything supports gzip. zstd, lz4, brotli, and others aren't included by default in most places yet, although the situation is starting to improve.

Plus, if you want super compatibility there is older-than-dirt pkzip/infozip.

it's optimal in that it's a community project and not an extremely unethical BigCorp project

Are you suggesting that lz4 is unethical?

just that zstd points to Facebook

It's obsolete. DEFLATE format is limited to 32KB LZ window with huffman coding.

Zstd can use a much larger window (8MB recommended) and a much better entropy coder: https://github.com/Cyan4973/FiniteStateEntropy

Last I checked QAT was limited to 64KB backreferences and independent 128KB blocks.

Now you know why they are comparing 16KB payloads only.

Yeah, that is definitely a limitation with QAT. It isn't a great fit for larger data, as it can quickly lose compression ratio due to its smaller window. However, there is a lot of compression done on data that is <64KB. E.g. compression in RocksDB.

We also have some half baked ideas to combine a fast SW match finder that only looks for matches >64KB away, and supplements the matches that QAT finds.

Reply on news.ycombinator.com