It's pretty interesting that they didn't just introduce an RCE that anyone can exploit, it requires the attacker's private key. It's ironically a very security conscious vulnerability.
If this hadn’t had the performance impact that brought it quickly to the surface, it’s possible that this would have lived quietly for a long time exactly because it’s not widely exploitable.
But if that was the case they wouldn't bother with the key. It'd be a one-and-done situation. It would be a stop-the-world event.
Now it looks more like nation-state spycraft.
Why not? It's possible someone else could've discovered the exploit before the big attack but decided not to disclose it. Or that they could've disclosed it and caused a lot of damage the attacker didn't necessarily want. And they easily could've been planning both a long-term way to access a huge swath of machines and also biding their time for a huge heist.
They have no reason to not restrict the backdoor to their personal use. And it probably is spycraft of some sort, and I think more likely than not it's a nation-state, but not necessarily. I could see a talented individual or group wanting to pull this off.
https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.h...
https://www.mail-archive.com/xz-devel@tukaani.org/msg00568.h...
https://www.mail-archive.com/xz-devel@tukaani.org/msg00569.h...
While this doesn't prove nation-state involvement, it certainly borrows from a wide-ranging playbook of techniques.
If you, as a maintainer, value stability of not only your software but also your own mental health, it is entirely something you can be proud of to resist calls for new features, scope increases, and rapid team additions.
But it's hard to see the whole thing helping, and it is some combination of depressing and infuriating. I hope he's doing ok.
It's also interesting because it exploits the current zeitgeist when it comes to maintainers' responsibilities to their users and communities. Personally, I think it puts too much expectation on maintainers' shoulders, especially when they're working for free.
Personally, I think maintainers are doing favors for users, and if they don't like how the project is progressing or not, then too bad. That's not a popular sentiment, though.
It seems like they were actively looking[0] which open source compression library they can inject with vulnerable code and then exploit and backdoor afterwards.
https://github.com/libarchive/libarchive/pull/1609#issuecomm...
> "In 2021, JiaT75 submitted a pull request to the libarchive repository with the title ‘Added error text to warning when untaring with bsdtar’ which seemed legitimate at first glance. "
Seems “Jia” is related to both
It could just as easily be a lone wolf capable of writing in different styles (really not difficult if one spends enough time online, one gets exposed to all sorts of styles) or even just your run of the mill criminal gang.
In fact, "multiaccount" cheating (where a single person has multiple accounts that interact with one another in a favourable manner, e.g. trading for stuff for reduced fees, or letting themselves be killed to pad stats, or expressing opinions leaving the impression of mass support, etc.) has been present in many MMO games for more than a decade. I recall an online game I was really into back in high school, eRepublik, which simulated real world politics, economics and warfare (in a really dumbed down version of course), and multiacounts were especially prevalent in politics, where candidates for election would often be padded by fake groups, or even entire countries being taken over by a gang of real people with like ten accounts each.
The complexity is nothing new. The only indicator this could be a state actor is the long con aspect of it. The attacker(s) were looking into years between starting and actually being able to exploit.
https://www.mail-archive.com/xz-devel@tukaani.org/msg00556.h...
That was not necessary to gain trust. Writing style is different, too. Later when Jia gained commit access he reminds him to merge it.
I think the most convincing case made about the sock puppets is around account creation dates, and also people disappearing after they get what they need. Like Jigar disappearing after Jia becomes maintainer. Or the guy "misoeater19" who creates his debian bug tracker account to say that his work is totally blocked on needing xz 5.6.1 to be in debian unstable.
It's literally a couple hours work.
Theoretically, I could even imagine that public key security motivated by some "mostly benign" actor who fell in love with the question "could I pull this off?" dreading the scenario of their future hacker superpower falling in the wrong hands. It's not entirely unthinkable that an alternative timeline where this was never discovered would only ever see this superbackdoor getting used for harmless pranks, winning bets and the like. In this unlikely scenario, the discovery, through the state actor and crime syndicate activities undoubtedly inspired by the discovery, would paradoxically lead to a less safe computing world than the alternative timeline where the person with the key had free RCE almost everywhere.
That's to say an enterprise router or switch would likely have secured exploits put there by corporate and national security agencies, whereas open source exploits would benefit from the probable deniability.
"It seemed like a good idea at the time" is a pretty common story though. ;)
So, while writing the exploit might be a couple of hours work, actually pulling it off is quite a bit more difficult.
For many projects, that is enough to become the main contributor.
I disagree, plenty of exploits implemented WireGuard-like crypto key access to them.
The tradeoff is that, once you find it, it's very clearly a backdoor. No way you can pretend this was an innocent bug.
It looks like the whole contribution to xz was an effort to just inject that backdoor. For example the author created the whole test framework where he could hide the malicious payload.
Before he started work on xz, he made contribution to libarchive in BSD which created a vulnerability.
Comment in your second link:
https://github.com/libarchive/libarchive/pull/1609#issuecomm...
(a) not exploitable without the existence of a much higher-severity exploit -- sure you can clear the screen, but that's low impact
(b) possible to trigger on other extant paths; see https://github.com/libarchive/libarchive/issues/2107 so it seems nothing new was introduced
(c) kind of contrived to get to execute; you have to somehow fail to extract the archive but on the happy path you'll see a bunch of weirdly-named files
I think it's distinctly higher-probability that this change was just meant to build credibility for the GitHub account. The diff is a fairly trivial patch to a minor issue filed around the same time as the pull request.
It's obvious, basically no one knows what's going on in the _vast_ majority of code running out systems these days. And even if you know 99% the attackers only need to be right once
There's some plausible evidence here that they may've tried to use alter egos to encourage Debian to update the package: https://twitter.com/f0wlsec/status/1773824841331740708
I don't think it's plausible OSS-Fuzz could have found this. The backdoor required a build configuration that was not used in OSS-Fuzz.
I'm guessing "Jia Tan" knew this and made changes to XZ's use of OSS-Fuzz for the purposes of cementing their position as the new maintainer of XZ, rather than out of worry OSS-Fuzz would find the backdoor as people have speculated.
Had this not been discovered, the backdoored version of xz could have eventually ended up in the ubuntu version oss-fuzz uses for its docker image - and linked into all those packages being tested as well.
Except now there's an explanation if fuzzing starts to fail - honggfuzz uses -fsanitize which is incompatible with xz's use of ifunc, so any package that depends on it should rebuild xz from source with --disable-ifunc instead of using the binary package.
And if that's the case, it was sloppy of "Jia" to disable it in OSS-Fuzz and not do this:
``` __attribute__((__used__,__no_sanitize_address__)) ```
to the XZ source code to fix the false positive and turn off the compilation warning, no attention would have been drawn to this at all since no one would have to change their build script.
With or without this PR, it's very unlikely OSS-Fuzz would have found the bug. OSS-Fuzz also happens to be on Ubuntu 20. I'm not very familiar with Ubuntu release cycles, but I think it would have been a very long time before backdoored packages made their way into Ubuntu 20.
It was reported by an MS engineer who happens to be involved in another OSS project. MS is doing business with the US intelligence community, for example there is the Skype story: First, rumors that NSA offers a lot of money for people who can break Skype's E2E encryption, then MS buys Skype, then MS changes Skype's client to not be E2E encrypted any more and to use MS servers instead of peer to peer, allowing undetectable wiretapping of arbitrary connections.
But it's a quite credible story too that it was just a random discovery. Even if it was the NSA, why would they hide that capability. It doesn't take much to run a script to compare git state with uploaded source tarballs in distros like Debian (Debian has separate tarballs for the source and the source with Debian patches applied).
I view it as being OSS or postgresql dev that happens to work at microsoft. I've been doing the former for much longer (starting somewhere between 2005 and 2008, depending on how you count) than the latter (2019-12).
Perhaps you’re not familiar with what NSA historically stood for: Never Say Anything.
Let's say agency A has some scanning capability on open source software that detected this backdoor attempt by agency B. If they had gone public, agency B now knows they have this ability. So agency B will adjust their ways the next time and the scanning capability becomes less useful. While if agency A had told Microsoft to "find" this by accident, nobody would know about their scanning capability. And the next attempt by agency B would only try to avoid having the performance impact this first attempt had, probably leaving it visible to agency A.
no this engineer is world-known for being core PostgreSQL, a team with high standards.. unlike that company you mention
(2) they could just say they found it during some routine dependency review or whatever
Second, maybe a routine dependency review is how they _actually_ found it but they don’t want future people like this focusing too much on that otherwise they may try to mitigate, whereas now they may focus on something inane like a 0.5 second increase in sshd’s load time or whatever.
Basically "parallel construction". It's very possible it's what happened.
People love when there's some explanation that doesn't involve randomness, because with randomness looks like we don't have grasp on things.
Google actually had tooling that was detecting it, but he disabled check that would show it.
Google/Microsoft/NSA could just say they detected it with internal tooling and not disclose how exactly. Google and Microsoft would love to have credit.
https://www.cnx-software.com/2021/04/22/phd-students-willful...
If it had been successfully and secretly deployed, this is the sort of thing that could make your leaders much more comfortable with starting a "limited war".
There are shades of "Setec Astronomy" here.
It was a psyop to increase the scrutiny around OSS components.
Kidding. Mostly...
But given the amount of scrutiny folks are going to start putting into some supply chains... Probably cheaper to execute than most company's annual security awareness budgets cost.
one other point for investigation is if the code is similar to any other known implants? like the way it obfuscates strings, the way it detects debuggers, the way its setting up a vtable, there might be code fragments shared across projects. Which might give clues about its origin.
[1] https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01
Typically its only state agencies that will fund an operation with uncertain pay off over long periods of time. That type of patience is expensive.
Online criminals are beholden to changing market pressures and short term investment pressures like any other start up.
That said it's a lot of work - 2 years at least. It's an exploit that's so good that you'd have to use incredibly carefully - also because if/when it's discovered it's going to break everywhere.
I've read descriptions about how the NSA (and presumably other such agencies) and they're really careful. The first job is to make sure the target doesn't get confirmation that they are in fact a target. The second is that they always cover their tracks so the target doesn't know they were a target.
Criminals tend to do the first but almost never the second so a tool like this - while I'm sure they would love - isn't worth the amount of work it would take to develop.
Again - I'm not 100% on this but ... 40% ? say 20% criminals, 10% lone wolf?
If this is or becomes a widespread method, then anti-malware groups should perhaps conduct these scans themselves.
So if distinguishing infected machines was their intention they definitely over-egged it.
I guess you could also include the fact that they made it a very obvious back door rather than an exploitable bug, but that has the advantage of only letting you exploit it so it was probably an intentional trade-off.
Just think how many back doors / intentional bugs there are that we don't know about because they didn't make any of these mistakes.
If they determine the vulnerability is only exploitable by the NSA for reasons
such as computational resources, budget, or skill set, they label it as NOBUS
and will not move to patch it, but rather leave it open to exploit against current
or future targets.
If they didn't have the key though, then yeah it doesn't fit. Unless they can walk through ssh key security anyway. :)
I'd like to rule-out some state actors by the mere heartbreaking act of socially engineering and targeting a person in pain.
Not even considering the original maintainer may be a US citizen (idk).
It's practically a good backdoor then, crypto graphically protected and safe against "re-play" attacks.
You can just treat the entire thing as opaque and proxy everything to the host you're trying to compromise; as soon as you have an exploit string for a given host you can just replay it.
E: see RFC 4253, sections 8 and 10, and RFC 4252 for corroboration
But you're right: we've verified the hostkey by the time we send the special certificate. So there's no way to effectively replay this without access to the server keys. My original comment is incorrect.
I'm actually surprised there's no logging at INFO or higher after this succeeds, given that openssh typically logs preauth connection closes. But I guess the crutch is that we never log connection opens and we only really log failures in handshaking, and it's not like the backdoor is going to go out of its way to log the fact that it opened itself...
Computers don't know who's / what's good or bad. They're deterministic machines which responds to commands.
I don't know whether there'll be any clues about who did this, but this will be the poster child of "you can't have backdoors and security in a single system" argument (which I strongly support).
"It's not only the good guys who have guns."