Hacker News Reader

Comment by jeffbee | Hacker News Reader

I love the QAT libraries and I feel their abilities are overlooked. Intel also has the igzip library that does not even require QAT and it radically faster than zlib, which is handy in older applications where gzip is unavoidable despite its obsolescence.

The major downside of course is it is quite tricky to use this stuff in practice. In the cloud, you need a bare metal instance that exposes the QAT peripheral, and they are relatively scarce. And this whole generation of hardware is only just beginning to land in public clouds. For machines you own, you will need to scrutinize Intel's somewhat ridiculous product matrix in order to acquire a Xeon that has QAT.

View on HN

> older applications where gzip is unavoidable despite its obsolescence

Is gzip actually obsolete or are there just newer alternatives? gzip is still everywhere

The zlib implementation is not optimal in any way that I am aware of. You can get better compression with the same compute time, and you can get the same compression with less time, in every application I have seen, by using some other thing. I usually want to look at zstd, lz4, and brotli in addition to zlib.

That said, the igzip re-implementation is really good. If you can use it, igzip moves gzip closer to the optimal frontier.

The only place gzip wins is in compatibility. Everything supports gzip. zstd, lz4, brotli, and others aren't included by default in most places yet, although the situation is starting to improve.

Plus, if you want super compatibility there is older-than-dirt pkzip/infozip.

There's also the various zlib forks by cloudflare et al, if you want to go down that road.

But yes, lz4 and zstd would be what I'd recommend people use.

The code quality of some of the cloudflare changes is questionable. Optimizations aren't cleanly separated by commits, some of the licensing is dicey, etc.

zlib-ng tried to merge as much as they could from the cloudflare fork without getting into licensing issues.

Oh, sure, it's all exciting in that respect from some of those, but I was more pointing to the collection of forks as a whole as a thing to evaluate if you have to look at zlib.

I would just go for lz4 and zstd, really, at this point, but if for some reason gzip is necessary or fits your use case better, ...

(Though I'd really like to hear what that is.)

it's optimal in that it's a community project and not an extremely unethical BigCorp project

Are you suggesting that lz4 is unethical?

just that zstd points to Facebook

It's obsolete. DEFLATE format is limited to 32KB LZ window with huffman coding.

Zstd can use a much larger window (8MB recommended) and a much better entropy coder: https://github.com/Cyan4973/FiniteStateEntropy

Last I checked QAT was limited to 64KB backreferences and independent 128KB blocks.

Now you know why they are comparing 16KB payloads only.

Yeah, that is definitely a limitation with QAT. It isn't a great fit for larger data, as it can quickly lose compression ratio due to its smaller window. However, there is a lot of compression done on data that is <64KB. E.g. compression in RocksDB.

We also have some half baked ideas to combine a fast SW match finder that only looks for matches >64KB away, and supplements the matches that QAT finds.

QAT does support SR-IOV on platforms with IOMMU support, so you can at least delegate access to QAT across multiple VMs. I don’t think any public cloud providers offer this, though.

any idea how igzip's non-QAT path compares to zlib-ng[1]?

[1] https://github.com/zlib-ng/zlib-ng

You can use TurboBench [1] to benchmark igzip, zlib-ng and others.

Download TurbBench from Releases [2]

Here Some Benchmarks:

- https://github.com/zlib-ng/zlib-ng/issues/1486

- https://github.com/powturbo/TurboBench/issues/43

[1] https://github.com/powturbo/TurboBench

[2] https://github.com/powturbo/TurboBench/releases

It's fairly easy to check for yourself, but with local builds on a desktop CPU without QAT, compressing a 150MB JSON input, with igzip and minigzip (from zlib-ng) both producing a smaller output than gzip, gzip needs 1.187s, minigzip needs 0.665s, and igzip needs 0.313s.

This is great, I can think of several places where zlib is the bottleneck and would love to speed them up

Intel’s software is simply best in class (n=2).

Reply on news.ycombinator.com