Comment by twotwotwo | Hacker News Reader

Comment by twotwotwo | Hacker News Reader

Parent Full thread

For anyone who wants to try this, zstd -T0 uses all your threads to compress, and https://github.com/facebook/zstd has a lot more description. Brotli, https://github.com/google/brotli, is another modern format with some good features for high compression levels, which also has Content-Encoding support in web browsers. You might also want to play with compression levels (-1 to -11 or more, zstd's --fast=n); they may not map 1:1 to gzip's.

One reason these modern compressors do better is not any particular mistake made defining DEFLATE in the 90s, but that new algos use a few MB of recently seen data as context instead of 32KB, and do other things that would have been impractical in the 90s but are reasonable with modern hardware. The new algorithms also contain lots of smart ideas and have fine-tuned implementations, but that core difference seems important to note.

So if you have many connections, you will spend gigabytes of RAM on compression alone? In the age of VM and cloud computing efficiency is important again.

If you are concerned about RAM usage, there are some trade-offs available (slightly lower compression ratio), for more effective resource usage.

As used by HAProxy for gzip filter, from the Will Tarreau, creator of HAProxy: https://github.com/wtarreau/libslz

When compressing, window size is a tunable option. I think the compression level sets it implicitly, but it can be set explicitly using brotli -w or windowLog in the advanced zstd API (https://facebook.github.io/zstd/zstd_manual.html).

Decompressing many streams at once is more interesting. The decompression APIs do let you specify a max history-window size, but e.g. HTTP won't let you advertise a window size limit, so all you can do is error out if the response would take too much memory to decode. You can just not advertise support for new Content-Encodings at all if it's a concern.

Virtually every compression format utilizing LZ77 has a way to declare the window size used and reject any archive requiring too large window. This is even the case with the traditional DEFLATE.

In the age of cloud computing, compression can easily be offloaded to another layer. CDNs can do brotli compression for you for example.

Reply on news.ycombinator.com