Xzbot: Notes, honeypot, and exploit demo for the xz backdoor

I suspect the original rationale is about preserving the longevity of the backdoor. If you blow a hole wide open that anyone can enter, it’s going to be found and shut down quickly.

If this hadn’t had the performance impact that brought it quickly to the surface, it’s possible that this would have lived quietly for a long time exactly because it’s not widely exploitable.

If you think of it as a state sponsored attack it makes a lot of sense to have a "secure" vulnerability in system that your own citizens might use.

It looks like the whole contribution to xz was an effort to just inject that backdoor. For example the author created the whole test framework where he could hide the malicious payload.

Before he started work on xz, he made contribution to libarchive in BSD which created a vulnerability.

For real, it's almost like a state-sponsored exploit. It's crafted and executed incredibly well, the performance issue feels like pure luck it got found.

This is called NOBUS: https://en.wikipedia.org/wiki/NOBUS

Am I reading it correctly that the payload signature includes the target SSH host key? So you can't just spray it around to servers, it's fairly computationally expensive to send it to a host.

I wonder if anyone has packet logs from the past few weeks that show attempts at sshd - might be some incriminating IP addresses to start hunting with

It's a (failed) case study in "what if we backdoor it in a way only good guys can use but bad guys can't?"

Computers don't know who's / what's good or bad. They're deterministic machines which responds to commands.

I don't know whether there'll be any clues about who did this, but this will be the poster child of "you can't have backdoors and security in a single system" argument (which I strongly support).

IMHO it's not that surprising; asymmetric crypto has been common in ransomware for a long time, and of course ransomware in general is based on securing data from its owner.

"It's not only the good guys who have guns."

how are you going to sell it if anyone can get in?

Imagine how much the private key is worth on the black market

It’s got me suspicious of build-time dependency we have in an open source tool, where the dependency goes out of its way to prefer xz and we even discovered that it installs xz on the host machine if it isn’t already installed — as a convenience. Kinda weird because it didn’t do that for any other dependencies.

These long-games are kinda scary and until whatever “evil” is actually done you have no idea what is actually malicious or just weird.

A main culprit seems to be the addition of binary files to the repo, to be used as test inputs. Especially if these files are “binary garbage” to prove a test fails. Seems like an obvious place to hide malicious stuff.

They removed the repo so only the attackers had access to the code and know how.

Well, it's a POC against a re-keyed version of the exploit; a POC against the original version would require the attacker's private key, which is undisclosed.

Probing for vulnerable deployments over the network (without the attacker's private key) seems impossible, not non-trivial.

The best one could do is more micro-benchmarking, but for an arbitrary Internet host you aren't going to know whether it's slow because it's vulnerable, or because it's far away, or because the computer's slow in general -- you don't have access to how long connection attempts to that host took historically. (And of course, there are also routing fluctuations.)

Depends how closely the exploit mirrors and/or masks itself within normal compression behavior imo.

I don’t think GuardDuty would catch it as it doesn’t look at processes like an EDR does (CrowdStrike, Carbon black), I don’t think sysdig would catch it as looks at containers and cloud infra. Handwaving some complexity here, as GD and sysdig could prob catch something odd via privileges gained and follow-on efforts by the threat actor via this exploit.

So imo means only EDRs (monitoring processes on endpoints) or software supply chain evaluations (monitoring sec problems in upstream FOSS) are most likely to catch the exploit itself.

Leads into another fairly large security theme interestingly - dev teams can dislike putting EDRs on boxes bc of the hit on compute and UX issues if a containment happens, and can dislike limits policy and limits around FOSS use. So this exploit hits at the heart of a org-driven “vulnerability” that has a lot of logic to stay exposed to or to fix, depending on where you sit. Security industry’s problem set in a nutshell.

Sysdig released a blog on friday. "For runtime detection, one way to go about it is to watch for the loading of the malicious library by SSHD. These shared libraries often include the version in their filename."

The blog has the actual rule content which I haven't seen from other security vendors

https://sysdig.com/blog/cve-2024-3094-detecting-the-sshd-bac...

That entire product category is for the most part snake oil.

No that patch series is from Lasse. He said himself that it's not urgent in any way and it won't be merged this merge window, but nobody (sane) is accusing Lasse of being the bad actor.

Lasse Collin is not Jia Tan until proven otherwise.

The referenced patch series had not made it into the kernel yet.

Also it may br a coincidence but JiaT75 looks a lot like Transponder 7500 which in aviation means hijacked...

Careful, APT28 is pretty dangerous. They are merging their ops with APT29 these days, and I wouldn't wake the cozy bear if I were you.

Cookie guy?

When loading liblzma, it patches the ELF GOT (global offset table) with the address of the malicious code. In case it's loaded before libcrypto, it registers a symbol audit handler (a glibc-specific feature, IIUC) to get notified when libcrypto's symbols are resolved so it can defer patching the GOT.

ifunc

Those are probably kill switches to prevent the exploit from working if there is a terminal open or runs in a GUI session. In other words someone trying to detect, reproduce or debug it.

Could that be related x11 session forwarding (common security hole on the connectors' side if they don't turn it off when connecting to an untrusted machine).

Yes.. The RCE happens at the connection stage before anything is logged.

Yeah, but then you would have ssh traffic without a matching login.

Wonder if any anomaly detection would work on that

> Anything available and this potentially involved in the build should be human readable.

That's actually a good principle to adopt overall.

We should treat this attack like an air plane accident and adopt new rules that mitigate the chances of it being successfully carried out again. We might not be able to vet every single person who contributes, but we should be able to easily separate out noisy test data.

There are things you can't contractually wave away, especially in form contracts that the other side has no ability to negotiate (which is what software licenses amount to).

One of those things is going to be fraud: if the maintainer is intentionally installing backdoors into their software and not telling the user, there's going to be some fraud-like statute that they'll be liable for.

Yes. This would surely be prosecutable under the CFAA.

Honestly, if I were involved in this, I'd hope that it was, say, the FBI that caught me. I think that'd be the best chance of staying out of the Guantanamo Bay Hilton, laws be damned.

You are talking about the greatest exposed hack of the computer supply chain so far by a big margin. Laws can be made retroactively for this type of thing. It has implications that are beyond the legal system, as the threat level is way beyond what is typically required as a sniff test for justifying military actions. This was not an RCE based on identifying negligent code; this was a carefully designed trap that could reverse the power dynamics during military conflict.

Legality things, depends on jurisdiction, which may or may not depend on:

* where were the authors of the code, * where the code is stored, * who is attacked, * where are their servers, * who is attacking, * where are they based, * where did they attack from, * ...

IANAL but it seems very complicated from law perspective (we, humanity, don't have a global law)

Edit2: making a bullet point list is hard

I’m no lawyer, but it is at minimum tortuous interference.

Is there some weird loophole or something I'm missing here? Otherwise, I'm not exactly sure how hacking wouldn't be illegal.

As a sanity check, attempted crime is still crime, so not a lawyer but part B isn't exactly ambiguous here: https://www.law.cornell.edu/uscode/text/18/1030

In terms of a plausible business model - I mean that's conspiracy to commit crime, like of course that's illegal? And depending on the state actor and the target system - literally treason if it's a cyber attack as a part of a war? You can put whatever you want into a license or contract but obviously that doesn't let you override the actual law

I brought this up in an earlier thread and got heavily downvoted. https://news.ycombinator.com/item?id=39878227

Jia Tan's GitHub activity was mostly 10-18@UTC, consistent with Europe/Israel/Russia

> Other countries such as Israel are playing the long-con with very well engineered, multi-year software backdoors

What is this in reference to?

> The USA seems to have an affinity for hardware interdiction as opposed to software backdoors.

What are some examples?

This goes back to WWII. USA solves problems with manufacturing and money. Europeans relatively lack both, so they solve problems with their brains.

I mean, they’re just the high profile ones. China makes and ships a lot of hardware, and the US makes and ships a lot of software.

My completely unexpert opinion, informed by listening to all the episodes of Darknet Diaries, agrees with this. US intelligence likes to just bully/bribe/blackmail the supply chain. They've got crypto chops, but I don't recall any terribly sophisticated implants like this one (except Stuxnet, which was likely Israel's work funded/assisted by the US). NK isn't terribly sophisticated, and their goal is money, so that doesn't match either. Russia is all over the place in terms of targets/sophistication/etc because of their laws (AFAIK it's legal for any citizen to wage cyberwarfare on anything and everything except domestically), but this feels a bit beyond anything I recall them accomplishing at a state level. Israeli organizations have a long history of highly sophisticated cyberwarfare (Stuxnet, NSO group, etc), and they're good about protecting their access to exploits. That seems to fit the best. That said, saying "Israeli organization" casts a wide net since it's such a boiling hotspot for cybersecurity professionals. Could be the work of the government, could be co-sponsored by the US, or could just be a group of smart people building another NSO group.

This actor, and others like them, may have dozens or hundreds of these things out there. We don't know. This was only found accidentally, not via code review.

So who's started scanning open source lib test cases for more stuff like this?

Must be punching the air right now

As I understand it Ed448 was only recently added to openssh, so maybe it was chosen in order to evade detection by analysis tools that scan for keys (if such a thing is possible.)

I believe ed25519 offers 128 bits of security, while ed448 offers 224 bits of security. ed448 has larger key sizes too, which does seem like an odd choice in this case. Maybe it was chosen for obscurity sake (being less commonly used)?

I assume the operation has most likely been called off. Their goal was probably to wait until it got into stable distros. I doubt there is a large number of unstable Debian or Fedora Rawhide servers with open SSH in the wild.

If it hadn't been discovered for another month or so, then it would have appeared in stable Fedora 40, Ubuntu 24.4 and Debian, and then it definitely would have been exploited. Another year it would have been in RHEL 10. Very luck escape.

https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...:

>Lasse regularly has internet breaks and is on one at the moment, started before this all kicked off. He has posted an update at https://tukaani.org/xz-backdoor/ and is working with the community.

OpenSSH and Linux were not targeted/affected.

xz and the Debian distribtion of OpenSSH were targeted.

Just needs a fine grained enough set of parameters that are observed on a system, aka "monitoring", and flag anything that sticks out. No need for "AI" unless the parameters that are monitored have more of a (complex) pattern to them than some value +/- deviation or similar things. "AI" is, in an abstract way, a pattern matcher and replacer, but also a big cannon that is often not needed and simpler heuristics work faster and better (and are usually deterministic).

Anomaly, fraud, outlier detection...

AI has been trying to solve that for several decades and in my experience in production these systems usually end up being a set of expert written rules saying "if X is greater than Y, notify someone to check".

Jia Tan could mean literally anything as pinyin since it doesn't have diacritics for tone. 嘉坛 (Jiā tán) could be a totally plausible name, and so could 佳檀 (also Jiā tán). Chinese names can largely use any character that doesn't come with a really bad meaning

Jia Tan is absolute a possible real name. Most people commenting on this topic aren't even Chinese native speakers lol. Each pinyin can map to a lot of different Chinese characters because they are missing the tone of the character anyway.

Tan is a real Chinese surname, and could be mandarin or hokkien.

Either way the name is likely made up. Obviously the attacker wasn't going to just use their real names.

There are many real Jia Tans on LinkedIn

The name that keeps coming up is Jia Tan (https://github.com/JiaT75/) but we have no way of knowing if this is a real name, pseudonym, or even a collective of people.

I'd say Ned Stark Associates, Fancy Bear, etc.

This Guy

https://news.ycombinator.com/item?id=39877604

Encrypting the payload will allow you to get by more scanners and in general make the traffic harder to notice. Since the publicly available server code needs to be able to decrypt the payload there is no way to make it completely secure, so this seems like a good tradeoff that prevents passive naive monitoring from triggering while not being more complicated than necessary.

The only real improvement that I can see being made would be adding perfect forward secrecy so that a logged session couldn't be decrypted after the fact. But that would likely add a lot of complexity (I think you need bidirectional communication?)

This is a NOBUS attack - Nobody But Us.

By tying it to a particular key owned by the attacker, no other party can trigger the exploit.

That action would create extra noise.

The thing about port knocking is that if you're on a host where you don't have the ability to port-knock, then you're not able to connect.

This can turn into a footgun: you're away from your usual device, something happens and you desperately need to connect, but now you can't because all the devices in your vicinity don't have the ability to perform $SECURITY_FEATURE_X so that you can connect, and you're screaming at yourself for adding so much security at the expense of convenience.

This could happen as easily as restricting logins to SSH keys only, and not being able to use your SSH key on whatever host you have available at the time, wishing you'd have enabled password authentication.

Port knocking is stupid. It violates Kerckhoffs’s principle¹. If you want more secret bits which users need to know in order to access your system, increase your password lengths, or cryptographic key sizes. If you want to keep log sizes manageable, adjust your logging levels.

The security of port knocking is also woefully inadequate:

• It adds very little security. How many bits are in a “secret knock”?

• The security it does add is bad: It’s sent in cleartext, and easily brute-forced.

• It complicates access, since it’s non-standard.

1. <https://en.wikipedia.org/w/index.php?title=Kerckhoffs%27s_pr...>

> Stuff like this is why I like port knocking, and limiting access to specific client IPs/networks when possible.

Indeed: I whitelist hosts/IP blocks allowed to SSH in. I don't use port-knocking but I never ever criticized those using port knocking.

I do really wonder if people are still going to say that port knocking is pointless and security theatre: we now have a clear example where people who were using port-knocking were protected while those who didn't were potentially wide open to the biggest backdoor discovered to date (even if it's wasn't yet in all the major distros).

> ... does the entire internet really need to be able to SSH to your box?

Nope and I never ever understood the argument saying: "Port-knocking is security theatre, it doesn't bring any added security".

To me port-knocking didn't lower the security of a system.

It seems that we now have a clear proof that it's actually helping versus certain type of attacks (including source-code supply chain attacks).

Over the weekend, I added this to a common bit included in all my NixOS systems:

    -networking.firewall.allowedTCPPorts = [ 22 ];
    +networking.firewall.interfaces."tailscale0".allowedTCPPorts = [ 22 ];

I probably should have done this ages ago.

Port knocking is bit like a lazy person's VPN. You might as well get off your butt and install a vpn solution and use ssh via vpn. The time and effort is almost the same nowadays anyway. The chances of both vpn and ssh being exploited like this must be zero.

Worse, most corporate, public wifi, etc networks block all sorts of ports. So at home sure you can open random ports but near everywhere else its just 80 and 443. Now you can't knock. But your https vpn works fine.

Also a lot of scary stuff here about identity and code checkins. If someone is some contributor, how do we know if their creds havent been stolen or they've been forced via blackmail or whatever to do this? Or how many contributors are actually intelligence agents? Then who is validating their code? This persons code went through just fine, and this was only caught because someone noticed a lag in logins, which by then, is a running binary.

FOSS works on the concept of x amount of developer trust, both in code and identity. You can't verify everyone all the time (creds and certs get stolen, blackmail, etc), nor can you audit every line of code all the time. Especially if the exploit is submitted piecemeal over the years or months. That trust is now being exploited it seems. Scary times. I wonder if how FOSS works will change after this. I assume the radio silence for Theo and Linus and others means there's a lot of brainstroming to get to the root of this problem. Addressing the symptom of this one attack probably won't be enough. I imagine some very powerful people want some clarity and fixes here and this is probably going to be a big deal.

I wouldn't be surprised if a big identity trust initiative comes out of this and some AI stuff to go over an entire submitter's history to spot any potentially malicious pattern like this that's hard for human beings to detect.

Jump host running a different SSH server implementation or SSH over VPN seems a little more reliable.

There's a lot of solutions now where the host has an agent that reaches out instead of allowing incoming connections which can be useful (assuming you trust that proxy service/software).

One place I worked, we ran our jumphost on GCP with Identity Aware Proxy and on AWS with SSM sessions so had to authenticate to the cloud provider API and the hosts weren't directly listening for connections from the internet. Similar setup to ZeroTier/TailScale+SSH

My gut-feel is that it rides near the line between "defense in depth" versus "security through obscurity".

Going in through the main Internet connection might not be the only way in. Someone surfing on Wifi who visits the wrong website can also become a second way into your internal network.

Got any advice to easily set up port knocking?

Rather than port knocking, I prefer IP knocking. The server has several ip addresses and once a correct sequence of connection is made, the ssh port opens. Since so few know about IP knocking, it much safer than port knocking.

/s

I’d recommend upvoting and commenting on the important points and adding your own useful comments.

These efforts have deterrence value