Everything I Know About the XZ Backdoor | Hacker News Reader

Everything I Know About the XZ Backdoor

boehs.org

195 pointsinternetter30 comments

This mailing thread is quoted in the 2022 part on the page on the xz maintainer being pressured to add new maintainers for xz: https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h...

Seems like the usual story of some lone maintainer maintaining a popular project and losing their time and energy.

The post doesn't spell it out but I wonder if they implied there's a suspicion that the person doing the pressuring there is also another sock puppet of JiaT75 as some scheme of getting access to xz. That would seem particularly cruel; take advantage of a tired maintainer with mental health issues to use their project to smuggle security exploits to the world.

Regardless, be nice to people who are doing "unpaid hobby projects" your work depends on. Reading the thread made me sad.

> The post doesn't spell it out but I wonder if they implied there's a suspicion that the person doing the pressuring there is also another sock puppet of JiaT75 as some scheme of getting access to xz

You can get the user's e-mail by clicking the "Reply via e-mail to" on the page. It matches the <firstname><lastname><number>@protonmail.com of the other sockpuppets, and the PGP key for the account was made 1 day prior to their first communication on that mailing list.

This is an intelligence agency trying to add backdoors to archiving software... likely targeting other utilities as well with same MO

Doubt the target is archiving software itself - presumably the reason these libraries got picked is because they already have high penetration across many layers of the stack which would ensure the backdoor has wide coverage.

The backdoor targets OpenSSH. The reason it's added to xz is that because of a complex dependency chain, it ends up being compiled to build OpenSSH. As far as I can tell, the payload doesn't get deployed into anything else.

It's worrisome for sure.. the original maintainer mentions longterm mental health issues, "but also due to some other things"

My worry would be "other things" they didn't mention can include deliberate acts of sabotage by said unknown agency. Devs can have health issues or other problems come up with themself or family in their personal lives, but also intelligence agents can tamper with people covertly in different ways such as deliberately causing various kind of accidents or contaminations/poisonings.

In any case; they could only have to disrupt the developer's life for a few months to persuade them that they need to step down to put one of their confederates at the head of the project, I begin to worry for All developers' safety now if you are the sole maintainer of a key project critical system daemons may link against.

Dunno, seems too amateur. An intelligence agency should be able to come up with more plausible sockpuppet names and email addresses even if in this case it didn't matter.

Ah I see, I wondered where the heck is a reply e-mail, I didn't see any reply email spelled out anywhere and didn't notice the button, I thought maybe the archive site deliberately wanted to hide the email adderesses.

Kinda looks like everyone in the thread might be sock puppet? ...except for the xz maintainer. Oof.

Now I'm just even more sad. Dammit Internet.

Assuming the original maintainer wasn't in on the con, someone should check on him. Apparently, he already got issues and being gaslit, manipulated and deceived on this level for years, considering the potential consequences and harm caused/barely averted, the unwanted attention, possibly police investigation following... all that may be a bit much.

I hope he's doing alright.

He replied that he is in holiday and will check in after Easter. And that he and Jia's GH accounts are inaccessible. More details too, but that's the crux as I understand it.

Main ongoing thread:

Backdoor in upstream xz/liblzma leading to SSH server compromise - https://news.ycombinator.com/item?id=39865810 - March 2024 (744 comments)

Wow. Timeline-wise this appears to be a clearly coordinated attempt, unless I'm missing something.

The timeline suggests this is a patient hacker. The most patient of which are nation-states because they're not worried about cashflow and think in terms of years.

Makes one wonder how many instances of this goes unnoticed. Maybe this attacker got sloppy, but how many of them don't? We were lucky that one person noticed suspicious behavior and had the technical skills and patience to investigate, but how many of us don't?

Many:

https://www.theregister.com/2022/02/23/chinese_nsa_linux/

Undoubtedly they do. This is an example of why defence in depth is important. Things fail all the time for intentional and non-intentional reasons. A single point of failure can’t be allowed.

It wasn't sloppy. It was just luck that someone noticed a half a second extra latency on the second connection of a newly run sshd process and went down the rabbit hole. Had they just shrugged and moved onto more "important" tasks/deliverables, it would most likely have landed in production across the world.

I'm a tad reminded of https://xkcd.com/705/

We got so lucky here. We won't get lucky every time. We will have a massive breach one of these days.

Kinda like Clifford Stoll!

I don’t think it was luck. I think some people are so in tune with their systems that investigating an anomaly like this is a frequent occurrence. This particular anomaly just happened to have an explosive ending.

Yes, I have met Andres in real life and I can totally believe that he is that in tune with his system. He wrote that he found this while benchmark PostgreSQL and saw weird load from ssh. He does a lot of benchmarking of PostgreSQL patches.

But I would say it was also luck. If Andres hadn't been benchmarking on Debian Testing (or whatever system he found this on) this might have taken longer time to discover.

It may not sound sloppy if you are used to todays apps and websites but half a second is an eternity in CPU time. Half a second is also very much a significant amount of time compared to normal ssh connection times with low network latency - if not Freund then someone else would have noticed, complained and this would have eventually been investigated. The only luck part here is it taking less than two months for this to happen but the attacker could have prevented this avenue for detection entirely by optimizing the exploit not to slow down the ssh proces.

> Maybe this attacker got sloppy

this attacker did not get sloppy, they got control of a critical piece of free software, and got a still-not-fully-understood piece of malware in to, past whatever peer review we like to imagine we have, then got it uploaded into at least two critical linux distributions, past whatever peer review we like to imagine we have, and was only found, two years in to the operation, by pure luck and a very dedicated engineer.

It was actually found less than two months after it was introduced in 5.6.0. Now, the attacker maintained the project for 2 years before that, and we don't know what else they inserted, but the known vulnerability was only in a release since February.

They did get sloppy though - the injected code slows down sshd startup enough that someone noticed and investigated. This could have been avoided.

Regarding the libarchive change:

> This commit does not do what it says it does. All it does, in fact, is replace safe_fprint with an unsafe variant, potentially introducing another vulnerability. The code was merged without any discussion, and lives on to this day. libarchive should also be considered compromised until proven otherwise.

This is very far from the truth, from what I can tell. If you look at the PR, it does, in fact, add strerror(errno) to the end of the error line, as it claims in its description.

And the switch from safe_fprintf() to regular fprintf() for the archive_error_string matches the rest of the codebase, which is scattered with calls to lafe_errc() and lafe_warnc() with archive_error_string (forwarding to vfprintf()). If the contents of that string were considered sensitive to print out, then it would hardly be a new vulnerability. Especially compare the code in tar/write.c, which calls fprintf(stderr) to output the archive_error_string in precisely the same way as the PR in question does.

Note that safe_fprintf() has nothing at all to do with memory safety: its only purpose is to escape unprintable characters before printing them. Indeed, a stack overflow bug within the safe_fprintf() implementation was the subject of a CVE in 2022 [0].

[0] https://github.com/advisories/GHSA-c42q-jv3x-rq38

Thanks, in my haste I misread. I removed the part claiming it didn't do what it says, though I maintain the changes made simultaneously is very suspect as there is no apparent motivation

the motivation is probably "get foot in the door". the attacker also made a few documentation-only PRs in various repos, but having code PR will make him more creditable, and also would help to add more backdoors to libarchive in the future

and librachive is now part of Windows 10, being used as a universal archive decompression library in the Explorer

This might be the tip of the iceberg, especially if this is part of the work of a nation-state agency. Very persistent and clever though.

Some other thoughts: how many popular open source projects only have 1-2 maintainers? I guess there is a way to do a search. I assume 5+ is safer, if they are of different humans.

Reply on news.ycombinator.com