In 2022, if you care about compression performance at all, there's no reason to even consider gzip unless you have a specific need for it (e.g. you have to deal with some other thing that can't handle zstd).
In 2022, if you care about compression performance at all, there's no reason to even consider gzip unless you have a specific need for it (e.g. you have to deal with some other thing that can't handle zstd).
One reason these modern compressors do better is not any particular mistake made defining DEFLATE in the 90s, but that new algos use a few MB of recently seen data as context instead of 32KB, and do other things that would have been impractical in the 90s but are reasonable with modern hardware. The new algorithms also contain lots of smart ideas and have fine-tuned implementations, but that core difference seems important to note.
As used by HAProxy for gzip filter, from the Will Tarreau, creator of HAProxy: https://github.com/wtarreau/libslz
Decompressing many streams at once is more interesting. The decompression APIs do let you specify a max history-window size, but e.g. HTTP won't let you advertise a window size limit, so all you can do is error out if the response would take too much memory to decode. You can just not advertise support for new Content-Encodings at all if it's a concern.
Maybe RHEL v9 took it into the main repositories, would have to check.
Maybe obvious for many but still worth mentioning is that on Unix-like systems any external compression can be used by piping to stdout, no need to rely on built-in support. Especially relevant with BSD tar implementations. Example:
tar cfv - /path/to/files | lz4 > output.tar.lz4That's a not-greatly-documented technology that's included in some processors (including a bunch of Celerons, but mostly Xeons) and in PCIe cards.
They are used in firewalls, ssl terminators and storage systems, as they can provide Gbps of compression or cryptography.
See one of the older cards: https://www.intel.com/content/dam/www/public/us/en/documents...
Netgate sells one 8950 based card for it's firewalls: https://shop.netgate.com/products/netgate-cpic-8955-cryptogr...
STH had a report this summer: https://www.servethehome.com/intel-quickassist-parts-and-car...
This cards are expensive when bought new, but you can probably buy some of them cheaply in eBay. Please don't drive the prices up ;)
Besides, what "better" archiving format are you going to use? Tar clearly isn't one (the great... grandparent comment wouldn't be asking for alternatives if it were). "Defacto standard" ZIP files have a poor compression ratio. 7z feels strange to use in a professional setting or for long-term storage. RAR is a closed-source abomination. Are there any other "better" archiving formats that have wider (de)compressor adoption than ZIP+LZMA?
Other than that, tar is still frequently used for packing data to be recorded onto LTO tapes (even despite LTFS being a thing).
I've used it often to speed up transfers of many small files over SSH - scp always took a small amount of time before sending each file, and with many files, the wasted time blew up. Serializing all those files with tar and deserializing them on the other side turned out to be much faster, especially since tar works in a stream-like fashion. And it's convenient to type, too:
tar -c path/to/file1 path/to/file2 ... | ssh user@host tar -x