Hacker News Reader

Comment by jgb1984 | Hacker News Reader

jgb1984

Maybe it's time to move away from xz which has been criticized in the past as a not great format, has had maintainer issues for years, and now turns out to be infiltrated with security problems. Moving to zstd seems worthwhile.

View on HN

We could say something similar about other tools. Even something like OpenSSL had a severe security issue (Heartbleed) discovered on 2014 and which was introduced 2 years earlier by the maintainer himself through a Git commit which occurred, if I remember, on the middle of the night of 31.12.2011 or 01.01.2012 (which lead some people to speculate whether that was intentional since "everyone" is usually celebrating with his/her family at that time of the year)

People did indeed move to LibreSSL and s2n-tls after that one incident

And later on crypto/tls, etc.

The maintainer merged the patch, but the code change came from someone else IIRC.

Best would be to disable anything in sshd that can perform compression or decompression. There is no value in that capability, and way too much risk.

Agreed on the SSH compression, but it would be not relevant to this particular vulnerability because liblzma was never used in the SSH compression anyway.

I don’t really understand the connection between liblzma and sshd, it had something do with systemd, or…?

Apparently liblzma in the compromised tarball would search for sshd functions and replace them with their own versions. This normally doesn't happen for the reason I've described, but many distros tend to compile sshd with a plugin that makes use of libsystemd, which depends on liblzma and therefore sshd will be linked with liblzma in that case. Liblzma could have done anything else it wanted, so the limited reach of this vulnerability is actually a better scenario...

Yes.

The connection is that sshd calls code in libsystemd when running as a systemd service, to notify systemd that it finished starting. An unrelated functionality in libsystemd (my guess is that it's related to parsing journald logs, since they can use LZMA compression) uses the lzma library, so libsystemd is linked against it. That is, the chain is: sshd is linked against libsystemd, which is linked against liblzma. When loading the sshd executable, the dynamic linker will load all these libraries.

The backdoor was added to initialization code in liblzma (which runs whenever the dynamic linker loads that library). So that malicious code runs before sshd starts executing, and manipulates the dynamic linker so that it will do something (which is still being analyzed AFAIK) when some specific functions are called by the sshd executable.

Wow, thank you!

So it is wrong for such a fundamental service as systemd to depend on such complicated processes ... just as the anti-systemd people said. Except of course shellscripts and /bin programs are just as complicated, when you check.

It's more or less what the current last comment on the Debian thread is wondering about

"Is xz-utils going to be maintained? Will we want to keep in the archive an unmaintained low-level library - low-level as in, susceptible of getting pulled as a dependency in lots of places - and rely on it for components such as dpkg?"

The format is fine, and zstd would be a downgrade in terms of compression ratio (which is the main reason xz is chosen for its current use cases). The lesson to learn here is not about making technical improvements, it’s to do something about improving the way we manage and maintain core open source infrastructure.

Ref XKCD 2347.

No, xz was chosen mainly because zstandard didn't exist at that time [1]. Zstandard does almost completely replace those "expensive" compression formats by providing a comparable compression ratio but being much faster.

[1] https://lists.debian.org/debian-devel-announce/2011/08/msg00...

"almost" and "comparable", but xz (lzma) still compresses at a higher ratio, which is still a highly desirable feature.

If the decision was being made today, a good faith debate could be had over whether the distro packaging use case benefits more from faster decompression or smaller file size, but I suspect that, globally, the average PC can decompress xz faster than the average broadband connection can download it.

I will assume that the average effective broadband connection is about 20 MB/s, and both xz and zstd decompressions are faster than the download under this assumption. But there are lesser known compression algorithms that are more efficient in terms of compression ratio for this target decompression speed, like zpaq [1], which are rarely considered as alternatives. So multiple factors exist here and I believe zstd excels at pretty much every other factor compared to xz, making it a very likely choice.

[1] https://mattmahoney.net/dc/zpaq.html

The format is fine, except for https://www.nongnu.org/lzip/xz_inadequate.html

Yeah, that title is click bait. The format is perfectly fine for long term archival.

In fact, I'd wager that all the focus that article has put on the minor problems with the format has taken much needed attention away from the fact that the maintainership makes xz-utils inadequate for long-term archival...

That attack article puts forward a dubious scenario where parts of multiple copies of a file become corrupted in different locations and you need to recover the original using both of the damaged copies.

If you find that sort of thing happening to you a lot, then lzip might be the format for you. But I can’t say I’ve ever heard of that being a real recovery scenario.

A much saner approach to enabling recovery from corruption would be to keep external checksums and parity files.

It's not, zstd can match LZMA2 compression ratio, while being much faster.

Much faster: yes.

While matching LZMA2 compression ratio: no.

Reply on news.ycombinator.com