Hacker News Reader

Comment by ris | Hacker News Reader

I love this case, because every time someone cites it as a reason packagers shouldn't patch upstream software I get to point out that they had to go back over a decade to find an example of it going bad. Soon it's going to be two decades.

View on HN

The xz backdoor was caused by packagers patching OpenSSH. Just because it was caught you don't get to pretend it doesn't count.

That was one factor of multiple, and the malicious actor chose to exploit it that way. liblzma is in enough critical stuff that they could have chosen to exploit something different had it not been for that.

You're confused there. The xz backdoor made use of a Debian OpenSSH patch, but it wasn't "caused" by it. Without the patch, the malicious xz maintainer could have written a different backdoor without making use of the OpenSSH patch -- for example, since debian packages are compressed with xz, the backdoor could have modified the sshd binary while unpacking the next OpenSSH security update. That would have been slower (attacker might have needed to wait a long time for a security update), and more discoverable since the modified file would be persisted to disk; but it also wouldn't have caused the performance issues that ended up in the discovery of the backdoor.

It would be discoverable but only if you ran an additional hash to check the final binary after updating and checking with an out of band source what the hash of the binary should be.

How many people double check that apt actually updated the package to the right version, if it’s output is compromised?

this is such a spin! that bug was two years worth of James Bond-level insertions into a situation that "was caused" by systemd ! if you want to get creative in the rewriting of fact

This isn't about systemd. OpenSSH is one of the most (if not the most) security-critical program in the distribution. Many systems run with just ssh enabled. That's why you don't mess with it.

Which library pulled the vulnerability in is mostly irrelevant.

When the init system won't reliably start openssh, and insists the only fix is to patch, then blame the horrible init system.

And that was what happened with systemd.

sd_notify is for additional (useful) functionality, it would work fine without it. You can tell because it works on arch.

You'd think it woukd work fine, after all, init systems for half a century have worked fine without it.

But no. Newer versions of systemd have issues, and this was what systemd pushed. Just why do you think all these distros had the sane patch? For fun?

Arch would have ended up with it eventually. It wasn't Arch being prescient, Arch wasn't using the same systend version as Debian Unstable, and other distros bleeding edge branches.

Do you really think there was no other avenue? There are tons and tons of things that link against liblzma, including stuff that commonly gets run as root such as apt, udev, and grub.

I believe the xz backdoor relied on Debian patching OpenSSH with libsystemd to work.

_sigh_ the backdoor was found because Debian also made those patches. Nearly all major distros were affected. The reason why Debian made the news is because the researcher who found the issue was using Debian. Had he been using Ubuntu, Arch, Fedora... those would have been in the news instead.

According to Arch: "openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma. Arch does not directly link openssh to liblzma", so at least one of your examples is wrong. That specific vulnerability was not in Arch.

The xz package was potentially vulnerable (although not in reality because "the build script was configured to only inject the bad code in Debian/Fedora based package build environments", while this was a choice by the attacker, it's still true the vulnerability wasn't there), but patching OpenSSH made OpenSSH specifically vulnerable when used with a malicious xz install.

https://archlinux.org/news/the-xz-package-has-been-backdoore...

Right... I may have wrongly named Arch in my comment. Thanks for the correction.

I'm curious about Arch's claim that "the build script was configured to only inject the bad code in Debian/Fedora based package build environments". Were Debian and Fedora specifically targetted, and the other distros who also got affected just happened to use similar packaging routines, or is this claim a guess?

The malicious build script included heuristics to only include the backdoor if you were building a .deb or .rpm package (the Debian and Fedora formats respectively). Other distros would have been affected if they used the same packaging setup -- Ubuntu also uses .deb, for example, because it's based on Debian.

And some distros IIRC considered themselves "affected" if they ever used a malicious version of the code, just in case, even if the backdoor didn't actually get compiled in to their version.

It's odd to call RPM the "Fedora" format. It literally means Red Hat Package Manager [0]. Well, at least it used to, according to Wikipedia. :D

It's true that Red Hat now owns Fedora, but the adoption went the other way around.

[0] https://en.wikipedia.org/wiki/RPM_Package_Manager

Me too, especially since many packages on Arch begin by downloading the official .deb/.rpm

Does this happen for official packages or just the AUR? I was pretty sure that when possible they build the packages from source themselves rather than bundling pre-existing binaries, and I'd assume that they wouldn't likely have the sources in the final package. IIRC Debian doesn't even like to bundle the headers in with libraries in preference of splitting them off into a separate file, so I'd be kind of shocked if they put the entire source of each library in the .deb files they distribute.

Arch package maintainer here; we generally encourage building from source for packages provided by the official repositories. As far as I know, we only ship pre compiled binaries if there is no source available, i.e. commercial programs such as reaper.

Awesome, that's pretty much exactly what I had thought

>"openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma. Arch does not directly link openssh to liblzma", so at least one of your examples is wrong. That specific vulnerability was not in Arch.

This is such a weird formulation though, because "other distributions" apparently included insignificant parts of the linux landscape like Fedora (i.e. the testing variant of the RedHat world) and SUSE.

And if the three largest upstream distris in the linux world have this mistake, calling that "Well some distris, but screw mostly Debian" doesn't sound like a strong point.

I wasn't claiming Debian were somehow singularly at fault, the poster just specifically said Arch was also vulnerable, which wasn't true.

Two open source programs I work on have had bad patches added by packagers.

This one was special because it lead to a massive security issue, rather than just annoyed users who had bugs only because of packaging patches.

Reply on news.ycombinator.com