Backdoor in upstream xz/liblzma leading to SSH server compromise

If it were me I'd be doing damage control to clear my name if my account was hacked and abused in this manner.

Otherwise if I was doing this knowing full well what would happen then full, complete defederation of me and my ability to contribute to anything ever again should commence -- the open source world is too open to such attacks where things are developed by people who assume good faith actors.

There is no requirement to use your real name when contributing to open source projects. The name of the backdoor author ("Jia Tan") might be fake. If it isn't, and if somehow they are found to be innocent (which I doubt, looking at the evidence throughout the thread), they can create a new account with a new fake identity.

So the really strange thing is why they put so little effort into making this undetectable. All they needed was to make it use less time to check each login attempt.

It just requires the SSH port to be reachable unless there is also a callout function (which is risky as people might see the traffic). And with Debian and Fedora covered and the change eventually making its way into Ubuntu and RHEL pretty much everything would have this backdoor.

Also seeing this bug. Extra valgrind output causes some failed tests for me. Looks like the new version will resolve it. Would like this new version so I can continue work.

--

Wow.

(Edited for clarity.)

  xz 5.6.1 -> 5.4.6

I understand it's unlikely, but is there anything I can do to check if the backdoor was used? Also any other steps I should take after "brew upgrade"?

(or SHAs, etc.)

(EDIT: 5.6.0 and 5.6.1 ?)

(EDIT 2: Ooof, looks like the nix unstable channel uses xz 5.6.1 at this time)

I use Nix to manage this stuff on Mac, not Homebrew...

1. They don't have the ability to freeze repos (i.e. would require some engineering effort to implement it), as I've never seen them do that before.

2. Many distros (and I assume many enterprises) were still linking to the GitHub releases to source the infected tarballs for building. Disabling the repo prevents that.

The infected tarballs and repo are still available elsewhere for researchers to find, too.

https://git.tukaani.org/

The current situation is ridiculous - if I pull in a compression library from npm, cargo or Python, why can that package interact with my network, make syscalls (as me) and read and write files on my computer? Leftpad shouldn’t be able to install crypto ransomware on my computer.

To solve that, package managers should include capability based security. I want to say “use this package from cargo, but refuse to compile or link into my binary any function which makes any syscall except for read and write. No open - if I want to compress or decompress a file, I’ll open the file myself and pass it in.” No messing with my filesystem. No network access. No raw asm, no trusted build scripts and no exec. What I allow is all you get.

The capability should be transitive. All dependencies of the package should be brought in under the same restriction.

In dynamic languages like (server side) JavaScript, I think this would have to be handled at runtime. We could add a capability parameter to all functions which issue syscalls (or do anything else that’s security sensitive). When the program starts, it gets an “everything” capability. That capability can be cloned and reduced to just the capabilities needed. (Think, pledge). If I want to talk to redis using a 3rd party library, I pass the redis package a capability which only allows it to open network connections. And only to this specific host on this specific port.

It wouldn’t stop all security problems. It might not even stop this one. But it would dramatically reduce the attack surface of badly behaving libraries.

Enough to be cautious, enough to think about how to catch bad actors, not so much as to close yourself off and become a paranoid hermit.

We should also be asking ourselves if we are working on critical infrastructure. Lasse Collin probably did not consider liblzma being loaded by sshd when vetting the new maintainer. Did the xz project ever agree to this responsibility?

We should also be asking ourselfs if each dependency of critical infrastructure is worth the risk. sshd linking libsystemd just to write a few bytes into an open fd is absurd. libsystemd pulling in liblzma because hey it also does compressed logging is absurd. Yet this kind of absurd dependency bloat is everywhere.

My customers and business are not any less important or valuable than anyone else's, and I should not be left being potentially exploited, and my customers harmed, for 90 more days while the big guys get to patch their systems (thinking of e.g. Log4J, where Amazon, Meta, Google, and others were told privately how to fix their systems, before others were even though the fix was simple).

Likewise, as a customer I should get to know as soon as someone's software is found vulnerable, so I can then make the choice whether to continue to subject myself to the risk of continuing to use it until it gets patched.

Is there "a new fixed release" ?

But why? A year is a ridiculous time for fixing a vulnerability even a minor one. If a vendor is taking that long its because they don't prioritize security at all and are just dragging their feet.

I would definitely be looking at every single commit though and if it isn't obviously safe I'd be drilling in.

Damage wise, most orgs aren't going to be hurt much by NSA or the Chinese equivalent getting access, but a Nigerian criminal gang? They're far more likely to encrypt all your files and demand a ransom.

randomly reverting two years of things across dozens of repositories will break them, almost definitely make them unbuildable, but also make them unreleasable in case any other change needs to happen soon.

all of their code needs to be audited to prove it shouldn't be deleted, of course, but that can't happen in the next ten minutes.

I swear that HN has the least-thought-through hot takes of any media in the world.

Slight oversimplification, see https://bugs.debian.org/1068024 discussion.

Build systems can even have undefined behaviour in the C++ sense. For example Conan 2 has a whole page on that.

My TLDR is that I would regard all commits by JiaT75 as potentially compromised.

Given the ability to manipulate gitnhistory I am not sure if a simple time based revert is enough.

It would be great to compare old copies of the repo with the current state. There is no guarantee that the history wasn't tampered with.

Overall the only safe action would IMHO to establish a new upstream from an assumed good state, then fully audit it. At that point we should probably just abandon it and use zstd instead.

So it's just odd that the tags and release tarballs diverge.

And I fail to see why bumping the epoch would ever be a problem. Using the epoch not a reason why its bad.

This particular backdoor is not shipped inside of a security patch, right?

"There's lots of stuff I have not analyzed and most of what I observed is purely from observation rather than exhaustively analyzing the backdoor code."

"There are other checks I have not fully traced."

If you have a large code base and organisation then keep doing those upgrades so it won’t be a problem when it really matters. If it’s painful, or touches too many areas of the code you’ll be forced to refactor things so that ceases to be a problem, and you might even manage to contain things so well that you can swap implementations relatively easily when needed.

Think about what various corps and state-level actors have been putting in there.

https://www.urbandictionary.com/define.php?term=Dickriding

I guess I'm not in the right demographic to know the term.

Jia had a zstd fork on github, but when things kicked off, it appears they may have sanitized the fork.

This is a very strong argument for FOSS to pick up the good habit of ditching/un-mainlining projects where they are sitting around for state actors to volunteer injecting commits to, and dep-stripping active projects from this cruft.

Who wants to maintain on a shitty compression format? Someone who is dephunting, it turns out.

Okay so your pirate-torrent person needs liblzma.so Offer it in the scary/oldware section of the package library that you need to hunt down the instructions to turn on. Let the users see that it's marked as obsolete, enterprises will see that it should go on the banlist.

All very fair points.

I've never understood how key-based systems are considered better. I understand the encryption angle, nobody is compromising that. But now I have a key I need to personally shepherd? where do I keep it, and my backups, and what is the protection on those places? how many local copies, how many offsite? And I still need a password to access/use it, but with no recourse should I lose or forget. how am I supposed to remember that? It's all just kicking the same cans down the same roads.

(But you are right that a sleeper would be affordable for them.)

Anyway. This team got caught. What are the odds that this state-actor that did this, that this was the only project / team / library that they decided to attack?

If there were a bunch of people who adopted it abnormally fast compared to usual, might point to there being more "bad actors" in this operation (said at the risk of sounding paranoid if this turns out to be a state run thing)

https://www.redhat.com/en/blog/urgent-security-alert-fedora-...

Any government which uses GNU/Linux in their infrastructure can pitch this as an attempt to backdoor their servers.

The real question is: will we ever even know who was behind this? If it was some mercenary hacker intending to resell the backdoor, maybe. But if it was someone working with an intelligence agency in US/China/Israel/Russia/etc, I doubt they'll ever be exposed.

So you review would need to guess from 2 new test files that those are, decompressed, a backdoor and could be injected which was never in the git history.

This was explicitly build to evade such reviews.

And? You never do any mistakes? Google "underhanded C contest"

In a large tech company (including ones I have worked at), sometimes you have policy where every change has to be code reviewed by another person. This kind of stuff isn't possible when the whole project only has 1-2 maintainers. Who's going to review your change other than yourself? This is the whole problem of OSS right now that a lot of people are bringing up.

I maintain a widely used open source project myself. I would love it if I could get high quality code review for my commits similar to my last workplace lol, but very very few people are willing to roll up their sleeves like that and work for free. Most people would just go to the Releases page and download your software instead.

Nation states will go to long lengths to disguise their identity. Using broken Russian English when they are not Russian, putting comments in the code of another language, and all sorts of other things to create misdirection.

It considers trust to be an individual metric instead of leaning more into the graph.

(There are other issues, e.g. the fact that "trust" isn't a universal metric either, but context dependent. There are folks whom you'd absolutely trust to e.g. do great & reliable work in a security context, but you'd still not hand them the keys to your car)

At least kuro5hin modeled a degradation of trust over time, which most models still skip.

It'd be a useful thing, but we have a long way to go before there's a working version.

Kuro5hin had a system which didn't work at all, as you mentioned.

But the best was probably Raph Levien's Advogato. That had a web of trust system which actually worked. But had a pretty limited scope (open source devs).

Now everyone just slaps an upvote/downvote button on and calls it a day.

1. Parse command line arguments

2. Setup logging

3. Load configuration files

4. Load keys/certificates into memory (notably including private keys)

5. Listen on a socket/port for incoming connections

6. Spawn a child process with reduced permissions (on Linux, using seccomp filters [2]) to respond to each incoming connection request

This backdoor executes at order 0 before sshd's main function is invoked, overwriting internal sshd functions with compromised ones. As some ideas of what the backdoor could achieve:

1. Leak server private keys during handshakes with users (including unauthenticated users) allowing the keys to be passively stolen

2. Accept backdoor keys as legitimate credentials

3. Compromise random number generation to disable perfect forward secrecy

4. Execute code on the host (supplied remotely by a malicious user) with the 'root' permissions available to sshd upon launch. On most Linux distributions, systemd-analyze security sshd.service will give a woeful score of 9.6/10 (10 being the worst).[3] There is essentially NO sandboxing used because an assumption is made that you'd want to login as root with sshd (or sudo/su to root) and thus would not want to be restricted in what filesystem paths and system calls your remote shell can then invoke.

The same attacker has also added code to Linux kernel build scripts which causes xz to be executed (xz at this point has a backdoor compiled into it) during the build of the Linux kernel where xz compression is used for the resulting image. Using this approach, the attacker can selectively choose to modify certain (or all) Linux kernel builds to do some very nasty things:

1. Leak Wireguard keys allowing them to be passively intercepted.

2. Compromise random number generation, meaning keys may be generated with minimal entropy (see Debian certificate problem from a few years ago).

3. Write LUKS master keys (keys used by dm-crypt for actually decrypting disks) to disks in retrievable format.

4, Introduce remote root code execution vulnerabilities into basic networking features such as TCP/IP code paths.

[1] 'main' function: https://anongit.mindrot.org/openssh.git/tree/sshd.c

[2] https://anongit.mindrot.org/openssh.git/tree/sandbox-seccomp...

[3] https://github.com/gentoo/gentoo/blob/HEAD/net-misc/openssh/...

Also, we only catch the ones that we ... catch. The ones that do everything perfectly, unless they come out and confess eventually, we don't get to "praise" them for their impeccable work.

then try to understand the pattern. they backdoored by modifying the build process of packages. now consider the $XZ is also from a backdoored build and the call recognizes in the same way with parameters --robot --version and the shell environment with the hint "xz_wrap.sh" from the piped process. a lot stuff to recognize for the $XZ process that it run as part of a kernel build.

Maybe they put advanced stuff in a backdoored $XZ binary to modify the kernel in a similar way they modified lzma based packages in the build process.

I would go further than that: all files which are in a distributed tarball, but not on the corresponding git repository, should be treated as suspect.

Distributing these generated autotools files is a relic of times when it could not be expected that the target machine would have all the necessary development environment pieces. Nowadays, we should be able to assume that whoever wants to compile the code can also run autoconf/automake/etc to generate the build scripts from their sources.

And other than the autotools output, and perhaps a couple of other tarball build artifacts (like cargo simplifying the Cargo.toml file), there should be no difference between what is distributed and what is on the repository. I recall reading about some project to find the corresponding commit for all Rust crates and compare it with the published crate, though I can't find it right now; I don't know whether there's something similar being done for other ecosystems.