Zlib – a spiffy yet delicately unobtrusive compression library | Hacker News Reader

Zlib – a spiffy yet delicately unobtrusive compression library

github.com

46 pointsmaydemir23 comments

I'm never quite sure about the state of zlib. There are commits going in [0] but at the same time pull requests with fixes aren't seeing much traction (e.g. [1]).

Optimization work happens in forks like Intel's [2], Cloudflare's [3], and Chromium's [4], the latter of which also has a lot of added tests and fuzzers [5]. And there's zlib-ng [6] which tries to create a home for many of the improvements, but is at the same time yet another fork.

[0] https://github.com/madler/zlib/commits/develop

[1] https://github.com/madler/zlib/pull/525

[2] https://github.com/jtkukunas/zlib

[3] https://github.com/cloudflare/zlib

[4] https://source.chromium.org/chromium/chromium/src/+/main:thi...

[5] https://source.chromium.org/chromium/chromium/src/+/main:thi...

[6] https://github.com/zlib-ng/zlib-ng

(edit: line breaks)

Going away from Zlib proper but staying with its format, there’s libdeflate[1], an implementation that’s substantially faster on modern machines but can’t stream (although I see no reason why it couldn’t, probably just for simplicity due to its origins as a baseline fast compressor[2]), and of course Zopfli[3], a glacially slow compressor that’s somewhat (and occasionally a lot) better.

[1] https://github.com/ebiggers/libdeflate

[2] https://encode.su/threads/2482-libdeflate-a-new-optimized-DE...

[3] https://github.com/google/zopfli

I have tried using zlib-ng on a project before, but I found it had too many issues when trying to deploy it. Almost every distro packs zlib, and zlib is so fundamental you can't just wholesale replace it, so you wind up with an almost the same library and regular zlib. I can't remember the exact details(maybe something with RPATH?), but we had to wind up just using the system zlib due to deployment issues.

The algorithm behind zlib is showing it's age, after all rfc1950 is 25 years old, and the deflate algorithm dates back to 33 year old pkzip. By now you get better compression at better speed from zstd, or vastly better compression at slower speed from lzma2, for basically anything you throw at it. But we still use zlib nearly everywhere, not least because it just works, no matter what language or hardware you use. It's a great piece of software and probably one of the most ubiquitous libraries out there.

LZ dates back 45 years but is still high performance. I can only imagine that we've been saddled with deflate because it's computationally light.

Well we have computation now and still enjoy using storage efficiently, so are (slowly, lazily) moving off deflate.

"LZ" is a vague family of algorithms, with a massively wide spectrum of performance and efficiency. Nearly every single modern compression algorithm falls into this family, with the only remotely popular exceptions I can think of being bzip2 and PPMd.

Is zlib the most installed software component in the world? Or is it curl or sqlite.

Interesting blog post about this topic:

https://daniel.haxx.se/blog/2021/10/21/the-most-used-softwar...

Next week on HN: “curl: A command line tool and library for transferring data with URL syntax”.

Anyway, the submission led to several very interesting technical comments so I don’t hate it.

I feel obliged to point this out here:

Do not build zlib with the hand-optimized assembly code.

The ASM code is a contribution and does not receive any support by the library authors. There are bugs in there and it will read/write out of bounds given just the right data - we've been bit by it. Note also [0].

The default Makefile gets this right - it stays far away from any hand-rolled assembly (or contributions in general). But the contributed Visual Studio solutions default to building with the ASM: [1] [2]. Make sure to choose ReleaseWithoutAsm, or set up your own configuration!

You can check any zlib build for whether it was built with ASM via (zlibCompileFlags() & (1 << 9)): [3]

Needless to say, an OOB in a library as fundamental (and as user-data-facing) as zlib is a serious vulnerability. I didn't find any distribution or major software that uses a vulnerable zlib build in a day of investigation (well, I think Telegram Desktop in a debug build might be vulnerable [4]), but if you are on Windows you might accidentally make one. Beware!

I'll also point interested readers towards zlib-ng [5] (without any endorsement or affiliation).

[0]: https://github.com/madler/zlib/blob/master/inffast.c#L12

[1]: https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da...

[2]: https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da...

[3]: https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da...

[4]: https://github.com/telegramdesktop/tdesktop/blob/6a3ad52aef0...

[5]: https://github.com/zlib-ng/zlib-ng

I'm surprised they don't rip it out if they don't want to spend the time to fix it. Or at least ensure nothing they provide builds it by default and you need to pass a `-DUSE_INSECURE_ASM` flag/`ReleaseWithInsecureAsm` in VS.

Well, the VS solutions (and the asm code) are both "contrib", so Mark Adler does not assume responsibility for either (which is fair - he has been stewarding this library for 30 something years already...). It could be made clearer that contrib does not come with the same implied fitness for purpose as the main library, but then open source licenses already disavow any and all such fitness in the first place...

It looks like the input word is using lodsd instruction, loading a 4 byte word, instead of a ptr increment, which could cause a 3 byte oob read if the last byte was at address 1.

I believe this is used in millions of builds, since it's been around for 20 years and Windows has a culture of winzip style software not packaged by Microsoft, but a lone programmer trying to get by with donations.

Do you have a specific exploit description? Mark Adler can remove or mark it unsafe on the github repo given a reason.

As ubiquitous as zlib is, I wonder why it’s not implemented in hardware like many audio and video codecs are.

It can be. But searching a large dictionary for the longest match is a pain. Audio and video codecs tend to have a much smaller working set or lots of parallelism for hardware depending on what stage of the process they are in.

It is implemented in hardware on IBM's mainframes, although not upstreamed in the main zlib repo [0]. It gives a massive speedup in my experience up to 20x for decompression and 220x (!) for compression

[0]: https://github.com/madler/zlib/pull/410

deflate is implemented in hardware on game consoles.

PS5, in addition to zlib, has hardware [0][2] decompressor for RAD Oodle Kraken [1], which is of course much better than zlib.

If at all possible, especially if you control both ends, please use zstandard[3] instead of zlib/deflate/gzip/zip for free general purpose compression. It is more advanced by about 30 years of technology development. It always spends less time to produce smaller files which are then decompressed insanely fast. There is no tradeoff, it is always better. It has a very wide range of compression levels you can choose from if the good default doesn't fit your need. It also supports multithreading, custom dictionaries and long range compression out of the box.

0 - https://www.tweaktown.com/news/71340/understanding-the-ps5s-...

1 - http://www.radgametools.com/oodlekraken.htm

2 - https://twitter.com/rygorous/status/1240341184867758085

3 - https://facebook.github.io/zstd/

> There is no tradeoff, [Zstd] is always better.

Well of course there’s the tradeoff in memory usage, utterly irrelevant on a desktop and probably even on a server, but you’re never decompressing a Zstandard file (compressed with standard options) on an STM32 (36 to 144 MHz ARM Cortex-M) micro or a similar RAM-impoverished but not entirely wimpy system. That is no secret, though, IIRC part of the original motivation for Zstandard was that modern hardware makes old window size choices obsolete.

A less obvious point is that a modern implementation of Deflate is, if not necessarily always faster, not as catastrophically slow compared to Zstandard as Zlib[1]. (In other benchmarks I’ve seen libdeflate be something like half again as slow as Zstandard, but at least it’s not multiple times slower.)

[1] https://lemire.me/blog/2021/06/30/compressing-json-gzip-vs-z...

If you are working on a microcontroller, odds are you will be keeping your entire file in RAM as that is all you have, in which case you should not need any memory for a window at all, as you can just use the file itself.

I don't know if the zstd implementation supports working like this, though.

I believe you can compress into the source buffer and keep the read offset ahead of the write offset, but I don't think you can decompress this way.

This is about the window buffer, not the source and destination buffers. They are still distinct, but you can combine the uncompressed buffer and the window buffer, if you have, or are going to have, the entire uncompressed data in memory.

Zlib is great. I used to use it all the time ~25 years ago. I don't directly use it much now but only because it has been a long time since I worked on anything where I had to deal with compression myself.

The only time I ever had a problem with it was on a project to implement a virtual CD-ROM drive for a client. The CD-ROM images needed to be compressed, and the client wanted the user of the ripper to be able to choose between more compression and more speed. Zlib supports different compression levels, but the speed increase from less compression wasn't as much as the client wanted.

I found an implementation of zlib that purported to be faster due to hand optimized assembler code. I tested that and it was faster, perhaps enough that the client would find it acceptable. But we would need a commercial license for it, and it was open source.

I contacted the author and he was willing to sell a commercial license to allow us to use it in all our products (generally utility software for Windows and Mac). We had an acceptable price worked out and were about to conclude the deal when he got cold feet. He started thinking that since we wrote Windows software and were a small company, maybe Microsoft would decide to acquire us, and then Microsoft would have a commercial license for his zlib.

He wanted to do a contract where the license would terminate if the company changed ownership. That would require getting my boss (the founder and CEO of the company) involved to work out the details and he didn't have time for that. I tried doubling the amount of money we were offering for the license to see if that would change the library author's mind but it did not.

My boss told me to find another solution, and to find one quickly because this was an urgent project for the client (for a completely stupid reason [1]). I found one.

25 years later I still can't decide if this was the most awesome hack I ever did or the most profoundly shameful thing I've ever written.

I added a slider to the ripper UI that went from 0 to 100. As the ripper ripped sectors, it either stored the sector in the image compressed with zlib at maximum compression level or it stored the sector uncompressed. The slider controlled what percent of the sectors were compressed.

The client was thrilled with this.

[1] The client was a major software company in Japan. Call them X. There was a small software company Y whose products did not overlap with those of X. X thought that a press release for a new product of Y's plagiarized one of X's press releases, were greatly offended by this, and so decided that they wanted to smash Y by directly competing with one of Y's best sellers. That was a virtual CD-ROM, so X told us to put the several other things we were doing for them on hold and quickly provide them with a virtual CD-ROM product and make it better than Y's.

Object not found! on the rfc doc linked in the README. What a shame RFC docs are so badly shared online this days. Shame on us!

Looks like the link format changed. https://datatracker.ietf.org/doc/html/rfc1950

Reply on news.ycombinator.com