FAQ on the xz-utils backdoor | Hacker News Reader

FAQ on the xz-utils backdoor

gist.github.com

141 pointsjethro_tell30 comments

Main discussion thread here: https://news.ycombinator.com/item?id=39865810

Backdoor in upstream xz/liblzma leading to SSH server compromise (openwall.com) 1848 points by rkta 5 hours ago | 654 comments

> The payload only activates if the running program has the process name /usr/bin/sshd. This means that systems that put sshd in /usr/sbin or another folder are not vulnerable. This further suspects targeting systemd systems due to their usrmerge initiative putting all binaries in /usr/bin.

This is highly likely wrong. The hack targets rpm and deb distros and as far as I know _all of them_ put sshd in /usr/sbin. The original oss-security report specifically said arg0 had to be /usr/sbin/sshd and I can reproduce his findings about increased runtime if you do so.

This /usr/bin vs /usr/sbin also has nothing do with usrmerge.

This type of mistake casts doubt in the rest of analysis.

There is no rational reason to justify throwing out the entire analysis over one mistake.

I said "Casts doubt". In the same way a typo in the headline, a logic error in the first bullet point, a misconception in the introduction, etc. casts doubt on the rest of a publication.

And this was no simple typo, as there was an accompanying and erroneous justification (which btw also hints to a lack of knowledge in the topic). This bullet was practically the easiest to verify out of the entire article and was already showing a contradiction. You might enjoy publications where fact-checking is not high in the list of priorities of the author (this article contains a disclaimer saying as much), but most "rational" readers should be interested in knowing how light the fact-checking indeed was, specially in a topic as important as this.

> The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf.

This doesn't seem really vital to the attack, but it's a silly outdated convention. Nobody is manually downloading and compiling tarballs these days, and those who do can figure out how to use autotools.

Yeah terrible idea. Either you should commit the output of autoconf (and enforce keeping it up to date via CI), or much much better - stop using autoconf! It belongs in the 90s!

It's another case of that "yes, it sucks. But the alternatives just plain don't work, or work in WAY fewer circumstances".

If I have a strange architecture or install, autoconf is often the only one that works.

What strange architecture are you talking about? I don't think the cost of supporting dead architectures is worth it.

CMake is a clear better choice (it also sucks but it sucks waaaay less).

I didn't want to start shitting on alternatives, but cmake is in my experience clearly the worst of all options.

It seems that more often than not, cmake setups work on the developer's own machine and nothing else. And the choice of undocumented "-DSOMETHING" is about as user hostile as it gets.

Reminds of the note at the bottom of Fish releases. It's there because the build system cannot determine the current version for some reason. Hopefully that will go away now that they have switched to a different language / build system. The custom tarball is used by Arch Linux at the very least.

https://github.com/fish-shell/fish-shell/releases/tag/3.7.1

https://github.com/fish-shell/fish-shell/issues/7772#issueco...

https://gitlab.archlinux.org/archlinux/packaging/packages/fi...

Yes, and in fact, it's quite a huge pain even in the packaging side.

Everytime I have to deal with autotools in nixpkgs, I end up having to rewrite the derivation to build from actual freaking source instead of a pre-mangled non-release tarball.

In fact, this makes me want to RFC *banning* release tarball usage.

you mean like LLVM’s source release tars? yeah, who would do that, haha?

So the attacker put the payload into the tests folder of the compression library and injected it upon build via some arcane build script? Kudos. That's quite a smart way to hide your backdoor.

Nevertheless, they got caught. So big respect for the analysis. Amazing work.

That being said, what does this attack tell you about the future of software security? Free software could easily be compromised by having a highly paid team create such a payload and then convince some maintainer, one way or another, to deploy it.

How do we back off from that cliff?

Security is hard.

In this case, it seems like there's a lot that could be done to reduce the chance of being exploited.

Why are obscure binaries in the repository at all in the first place, even if in the test dir? Surely those binaries can be generated as a setup step prior to the tests actually running.

Furthermore, why are there scripts present in the release package that have not been added to the source code?

Also, if I understood it correctly, the compromised binaries are also present in the release tarball. Why are they part of the release if they are supposedly test fixtures rather than part of the application code?

> Why are they part of the release if they are supposedly test fixtures rather than part of the application code?

Test fixtures should be distributed together with the application code, so that the actually built application can be tested. Ideally, the application should be tested every time it's built, to ensure that for instance a change in the compiler being used didn't break it.

Thanks, that makes sense.

I think in my mind the whole chain was still

    source code->compile->test->distribute

from a single repo which is obviously not the case.

It’s more like

    source->package->test->ship to distros->compile->test->package->distribute

To be fair, so could closed source code.

The difference in this case is that a guy was running perf tests against postgres and noticed his ssh was slow, so he dove into that and found the backdoor a couple days after it was pushed.

If you had to submit 'my ssh daemon is slow' to a third party it would get put on the pile with the rest of them

Hmm, well they never find these kinds of attacks in closed source code. OTOH they never find these kinds of attacks in closed source code!

Closed source is a little tougher: Often you need to get a job somewhere in their organization or supply chain.

Or find a security vulnerability in their network, say a poorly secured FTP server as per SolarWinds.

And then you wouldn't have random people discovering it! It could even be covered up indefinitely.

People are prematurely celebrating this did not reach a stable distribution. But we haven't seen the full fallout yet. How many debian-developer's machines have been compromised, for example?

I don't think that's the biggest concern.

There are 2 years of commits that potentially have backdoors.

So wait, if I did a find / -n 'liblzma*' and saw 5.4, I'm good? This is only 5.6 and 5.6.1, right?

From what is publicly known at the moment, yes.

Note however that xzutils home page says that "versions 5.2.12, 5.4.3 and later have been signed with Jia Tan's OpenPGP key" so there would have been plenty more opportunities. We may just have seen the beginning. Whoever did this played the long game.

Also note that there was proposed patches by this compromised project maintainer to oss-fuzz and valgrind to avoid the detection of this backdoor.

The attacker had 750 previous commits; maybe it was all for this or maybe there are more vulnerabilities.

so this backdoor is a package which installs a fake sshd? what is selinux for?

Flagged: How is this "via a project dev"? Which project? The guy doesn't seem to have anything to do with the discovery of this backdoor, and he just summarizes Andres Freund's announcement.

The author of this is a gentoo maintainer who focuses on security work. Most distros have a couple of point people handling this issue for them at this point, and they are all likely pretty well informed.

If you want clarification on something, you can just ask instead of announcing that you've flagged it.

Its from the Gentoo maintainer of the package and libarchive project dev that submits changes, patches, is involved in the mail list and has ops on the groups IRC chan.

It's included in Lasse Collin's FAQ now that he has been found, but in his absence, Sam was speaking for the project on IRC chan and filed the gentoo bug also linked at the bottom of in Collin's FAQ:

https://tukaani.org/xz-backdoor/

You could have just asked and I could have elaborated.

Reply on news.ycombinator.com