XZ backdoor story – Initial analysis
securelist.com
securelist.com
I think this sentence actually understates what happened.
What I find more frightening than the technical aspects of the backdoor is the amount and degree of "social engineering" involved: The backdoor was the final product, but inserting it was only possible because at that point the entire xz project had long been taken over by malicious actors ("Jia Tan" and friends) - after a yearlong psychological operation against the maintainer. All that without the maintainer or anyone else noticing anything.
That's spy novel stuff - and if something like this is possible, it makes me wonder what else might be going on in other projects right now.
(You can see the same mentality reflected in the backdoor code itself - I find it remarkable how much of the code is devoted to active misdirection. It's not just trying to look harmless, the code actively tries to build a "narrative" through commit messages, comments, variable names, choice of commands, etc, of what it's ostensibly doing while really doing something completely different - leading anyone who'd inspect the code first to doubt themselves and their understanding of the code, then suspect a bug and then, maybe, suspect malicious intent)
I hope some three letter agencies are conducting a deeper investigation into this.
I hope this can be a lesson learned. A lot is going on. There are lots and lots of agencies (both government-funded in all countries as well as black market ones) with a mission to own backdoors. They're all well-funded. That's what they do.
It is frustrating to watch every HN thread related to backdoors filled with rejections of this possibility as paranoia or silly tinfoil hatting. As if it wasn't happening. Here's one concrete case that was caught. Countless others, not yet caught.
And think about this one: this was on an open source project, which makes it easier to catch. Still we got lucky. Now consider all the closed source products, where backdooring just means infiltrating and/or arm-twisting a single organization. This happens often. Nobody wants to believe it but it's common. Anyone who has worked on tech infrastructure companies has some stories. They are difficult to tell due to NDAs and worse, but it happens.
I completely agree. What seems incredible to me is that how much manipulation, thoughtfulness, patience, perseverance went into this. It's either a product of the obsession of someone, or a private security firm's or state actor's doing, who do these things to multiple projects as part of a 9-5 job.
But I would love some deeper analysis of the mistakes and over-engineering parts. In the Bryan Cantril interview [1] Andrés suggests there are a lot of dumb things done because it was a kind of off-the-shelf backdooring piece that didn't necessarily know how it would be deployed. Eg the symbol table lookup that led him to investigate.
Similarly, why were they chopping off 48 bytes using RC4 [2]
So I'd love folks to talk about how (given more time/better team) this could have been better or where they messed up more
[1] https://youtu.be/jg5F9UupL6I?si=gvXsYFXgagkGOMd4 [2] https://twitter.com/matthew_d_green/status/17744729080201014...
That would improve the supply chain security of code getting linked in somewhere but never executed.
EDIT: Or even better, perhaps ifunc should be implemented in a declarative fashion so that you can't just cause arbitrary code execution from each library you link against. That might be harder to implement at this point considering back compat, but probably is something that could be layered in over a longer time period (i.e. if you build any library with "declaratively linked ifunc" feature bit set, then the dynamic linker will force all linked libraries to have that feature flag or fail to launch).
Currently building most libraries usually involves executing a series of highly complex byzantine scripts, requiring turing-complete environment. This gives attacker an endless attack surface, and when the build process is hijacked - well, the opportunities are there.
Moving to a declarative build process with only a limited state machine as an executor would help. Requiring all source blobs being reproducible might also be something to think about.
Don't limit the build system language. It makes builds more complex, not less.
People put trust in distros and packagers to having something of a review chain - there's 0% chance you personally be an expert in everything executing on your workstation right now (outside of maybe toy systems). I'm not an expert in m4 or bash, but I hope enough experts are in the chain to get to my distro's package library are that such things are less likely. But that is all bypassed here.
I think this particular approach is a one-off, as I know of no other build environment where it's expected to have the generated executable of the build system "helpfully" packaged in the tarball as a packaging step.
If it is in some I'm not aware of, I hope that decision is being re-examined now.
Instead it was hidden in plain sight in the source that is processed by autoconf.
> it's expected to have the generated executable of the build system "helpfully" packaged in the tarball as a packaging step.
While this is true, most distros as I mentioned above rerun autoconf to ensure consistency in the tests and possibly to include autoconf bug fixes.
The code to include the backdoor in the build was in an m4 script.
The initial reporting said that this code was not present in the github source, but the post-autogen code (including the attack) was included in the github releases.
The article says that this modified script was present in the source on tukaani.org, which was controlled by the attacker and used by the distros as their upstream.
If you downloaded from github and reran autogen you were OK. If you downloaded from tukaani and reran autogen, like the distros did, you lost.
better hygeine that way would also simplify both environments, since the tools for each are fairly different.
xz is not different in this respect from any other C, C++, Go or Rust program.
I mean the github repository with exactly the same .ac and .am files works fine with local automake/autoconf generation without that file existing. And thus no backdoor (the test files are still there, but "harmless" without the initial kicking off point to actually de-obfuscate and include their contents)
> if that is installed on the system is not required by the build
This specific file defines a macro that is used by autoconf, not by the build. If it is installed on the system it is not required by autoconf, but then gnulib is practically never installed.
Your original message blamed the backdoor on "the generated executable". This m4 file is not a generated file and not an executable. It is simply vendoring like you often see in other languages.
"Vendoring" by copying untracked files into the tarball seems discourteous to the definition. It seems to rely on the "possibly odd" behavior of autoreconf to allow files that happen to have the same name to override system-installed versions? I guess on the belief that local definitions can override them is useful? But that certainly bit them in the ass here. As to get a "completely" clean autoconf rebuild it looks like you have to delete matching files manually.
For example if you've ever taken a look at the bluetooth specs you would not trust a single person in the world to implement it correctly and you probably wouldn't even trust an arbitrarily large team to implement it correctly.
Unless they had a long demonstrated and credible track record of shipping perfectly functional products and an effectively unlimited budget, i. e. Apple and maybe 1 or 2 other groups, at most.
I messed around a tiny bit with Bluetooth on Linux recently. Going to rankly speculate that Bluetooth is such a special case of hell such that it makes a distracting example here.
I mean, as a joke suppose we wanted to design a 3.5 mm patch cord that pitch shifts down a 1/4 step for randomly chosen stretches. It turns out to be easy-- just remove the wire from the casing and replace it with cheap bluetooth chips at either end. You'll get that behavior for free! :)
Compare to, say, USB, where your point above would apply just as well. I wouldn't be distracted by that example because even cheapo, decades-old USB drives to this day let me read/write without interleaving zeros in my data.
Shit, now I'm distracted again. Does Bluetooth audio even have the notion of a buffer size that I can set from the sender? And I guess technically the receiver isn't interleaving the signal with zeros-- it's adjusting the rate at which it sends blocks of the received data to the audio subsystem.
Was Bluetooth audio basically designed just for the human voice under the assumption we're constantly pitch shifting?
Oops, I almost forgot-- the sample-rate shift is preceded by a dropout, so I do get interleaved zeros in the audio data! I actually get a smoother ramp in my Prius changing from battery to ICE than I do in my wireless audio system!!!
Anyhow, what were we talking about again?
Government funding for defense of the economy and computing in the "free world".
Certainly the defense department has a billion to spare, and so does the EU
I think it's important to push back on a very specific point here. It's true in general that if the attacker has added a backdoor in your library, and you're going to call that library's code, you've pretty much lost. Game over. Go home.
But this was a very different attack, in that the attackers couldn't directly target sshd. They couldn't manage to target any library that sshd calls directly either.
The library with the backdoor is code that was never actually called at runtime. This is important because it means it *doesn't* have a hundred ways to reach code execution. It only had a few select ways, mainly constructors and indirect function resolvers.
The amount of weirdness in glibc's runtime loader is not unbounded. There aren't actually a hundred places where it allows random libs to run code before main. And we should take a good look at those couple place that are clearly juicy high-value gadgets to attacker.
When glibc's runtime loader first loads a binary, first reaches a relocation for a STT_GNU_IFUNC, everything is still in a pristine state and no arbitrary code can run without being explicitly called. Attackers don't have magical powers that allow them to run code before you hand them the control flow. At this point in the runtime loader, an ifunc resolver cannot do anything without being caught. It cannot "just open /proc/self/mem" or "just call mprotect". It cannot "just disassemble the caller" or "just overwrite the GOT".
I really want to hammer home how different that is from letting the attacker run code after main. There's nothing you can do if you directly call a backdoored library. But we shouldn't let attackers get away with spending 400ms parsing and disassembling ELFs in memory in an ifunc resolver of all things. Overwriting the GOT like nobody's watching.
The backdoor isn't magic. For all the beautiful sophistication that went into it, it made many mistakes along the way that could have led to a detection. From valgrind errors to unacceptable amount of noise (400ms!) before main.
The danger with supply-chain attacks is that it could come from practically anywhere. Attackers can choose to target an overworked maintainer in a third-party library, and it's much easier for them than going after OpenSSH itself.
About the OpenSSH maintainers, they're known for being the paranoid amongst the paranoid. No one's infallible, but if attackers are forced to go directly after them instead of bullying smaller libraries, I'll have a reason to feel safer about the reduced attack surface :)
Defense in depth is about adding more & more hurdles for attackers which raises the costs involved. Even state actors have a budget.
It would be simpler to statically link sshd (or, more generically, all sensitive system binaries) and launch them in the R^X mode via, say, specifying the required capability either in the binary descriptor or at the system configuration level (SELinux) – to force-launch the process in the R^X mode for its code pages. The problem, of course, is that not every Linux distribution comes with SELinux enabled as a default.
Alas this won't work. Dynamic linking is lazy, there's no moment when it's "complete". The correct function pointers get loaded and inserted into the table (in place of stubs) when called the first time, which can be arbitrarily far into the future. In fact in most large library ecosystems (gtk apps, etc...) most of the linked functions are never called at all.
with lazy binding the function resolver might not even be called at all, the link tree is like sshd.elf -> systemd.so -> lzma.so. If systemd uses a ifunc symbol in lzma but sshd is not using that symbol: if lazy binding the resolver will never run, if eager binding the resolver will run. Something the backdoor also took advantage of
I know it was a pain to use correctly. But still.
If you don't trust a library, you can run it in a separate sandboxed process (or app domain, or wasm container, or whatever your language provides).
Even a Rust program with no unsafe code can do this.
Patching readonly libc code from Python without external modules is also trivial, resulting in arbitrary execution!
But the analysis of the binary in that detail seems new to me.
So how has the source code shown there been produced? Running a disassember, understanding what the code does and renaming everything to descriptive names? That looks quite like quite an achievement in 2 weeks.
>Global Research & Analysis Team, Kaspersky Lab
https://securelist.com/author/great/
The article is apparently authored by malware analysis team from Kaspersky Lab, so they are probably quite good at reverse engineering binaries.
The people who have reverse engineered the code say it also requires the command message to be bound to the ssh host key as well, so if the host key is an RSA key then it might be doing an extra RSA decryption operation on every connection as well?
That would probably do it.
(OpenSSL benchmark numbers)
sign/s verify/s
256 bits ecdsa (nistp256) 34642.6 11700.3
sign/s verify/s
456 bits EdDSA (Ed448) 3209.5 409.5
However yes, even on this, not so powerful CPU, it doesnt take 500ms so I dont think it explains it.
Curious that asymmetric crypto isn't even the slow part here. Feels like they just messed up somewhere (but I don't have the low-level asm/C skills to check that without considerable time investment)
The really went crazy with the home-made x86 disassembler and everything they do with it. Must have been super fun to write though! Someone clearly had a great time coming up with clever ideas for a backdoor.
IIRC the original detection was because sshd was using a lot more CPU coping with the torrent of spam connections any sshd on the internet gets than it usually would.
If some inetd-like program were listening on port 22 and fork+exec'ing sshd to handle each incoming connection, that would explain it. But on my systemd-based Linux system I see a long-running sshd that appears to be taking care of port 22.
I do agree that it seems like the best explanation of the delay is that somehow sshd was being exec'ed per connection, but I haven't seen all the dots connected yet.
You should also be able to see the error message "sshd re-exec requires execution with an absolute path" in the source. If you follow the `rexec_flag` tested above this message, you can see where it calls execv, later in the code.
Easy way to find out from outside if server is compromised without trying to inject code first.
The custom elf parser and disassembler are so complex that I can't imagine they didn't use that code / or will never use that code for something else in the future.
I wonder if this gets the serious investigation it deserves, but I doubt it.
The slowness derived from all the symbol and instruction lookups it did afaik
https://open.spotify.com/episode/0C04JlCpMFe07C1XCswbZ9?si=B...
How do you get one step ahead of that? Does the community focusing on SSHD impact some other part of the system as a whole? Some other technological aspect? Social?
Tin foil hats are fun!
IME/IIRC There aren't (or shouldn't be) any binary contents on Flathub that are submitted by the author, at least for projects with source available? You're supposed to submit a short, plain-text recipe instead, which then gets automatically built from source outside the control of the author.
> The Flathub service then uses the manifest from your repository to continuously build and distribute your application on every commit.
https://docs.flathub.org/docs/for-app-authors/submission/#ho...
Usually the recipes should just list the appropriate Git-based URLs to get the source code, or, for proprietary applications, the official .DEBs. Kinda like AUR, but JSON/YAML. Easy to audit if you want:
https://github.com/orgs/flathub/repositories
Plus, if properly configured, Flatpak is supposed to securely sandbox every application anyway, although other systems it needs to be compatible with (E.G. X11) currently prevent that.
And then the frequent use of git submodules makes it even worse: that's often a whole lot of code that even the author of the flatpak doesn't have control over.
The JSON manifest is a much smaller attack surface than uploading random binaries would be though. And the standardized build procedure should make it relatively easy to tell if something's out of the ordinary and should be raising eyebrows, or even automate much of it.
Maybe stick an `alias CheckFlatHub=` for a LLM prompt, or just some plain regexes, in `.bashrc`? Looking for fishing URLs and install commands sounds like a relatively simple problem, as far as security challenges go.
In fact we already have a very good solution, and it's simply the infrastructure of a typical Linux distro (replicated by package managers on other systems, such as F-Droid). It's quite an impressive feat really - a diaspora of independently trusted organizations, each with their own governance structures and PGP-enforced webs-of-trust, crafting public repositories of known-good software built from publicly published source code using publicly published build scripts, delivered to the user over strong encryption. On the whole this does an incredible job of keeping malicious or subversive software out, and we've all got so comfortable with just being able to "apt-get install <foo>" without worrying about malware that we've forgotten what a dangerous business running software from the internet used to be.
Which is why this xz attack is so unusual and surprising - it's like the police stopping an attempted murder in the nick of time in a sleepy crime-free town. As shocking as it is, we must ignore the cries of "something must be done" and resist the temptation to put up barbed wire fences all over our nice town.
For example, Ubuntu 24.04 will ship without Deno. It will also ship with Cargo 1.75, which is too old to build Deno (there's some weird packaging issue). So, anybody who wants Deno on Ubuntu has to install it using either `curl | bash`, nix (which of course seems promising), or asdf (which I know nothing about). Most devs would choose curl|bash.
Why is Deno not packaged for Ubuntu/Debian? Because it's "hard", and it's hard in different ways for different distros (the debian ticket for this has sat without action for 2-3 years, and I think the Deno ticket a similar amount of time). Cross-distro solutions like flatpak (or nix) are 100% necessary if we are to get rid of curl|bash. But, the sandboxing story with flatpak/snap is currently pretty bad; apps just fail in mysterious ways if they try to access resources they're not allowed access to. Nobody wants Windows Vista, but it has been done reasonably well in various other OSes.
That's a feature. I don't want anything bleeding edge. Give me older battle tested to hell and back code. Nothing new.
[1] https://github.com/tukaani-project/xz/commit/780d2c236de0e47...
Or buy closed source from someone like Microsoft, hoping that they have the resources and actually use the resources to scrutinise code more vigorously.
And there is always the approach of having a great secops team to detect strange network activity and attempts to escalate.
But from contributors who have no history and no footprint other then the project they're working on. That should be a red flag in the future.
The old groups GOBBLES,ADM,ac1db1tch3z,~el8 were doing it, private "security researchers" like isec.pl were doing it.
This time it's a problem because state actors are abusing the corporate capitalism that created this era of underpaid people working on foundational projects. The bad actors have unlimited resources for their objectives.
That's basically what created the demand and ineption of groups like NSO,Zerodium etc..
Basically before that exploits and backdoors were worthless and hackers hoped to be sponsored or hired by companies like Qualys.
And I don't see what money has to do with any of this. There could be some well-paid programmer slipping backdoors into proprietary code just as easily. The salary or openness didn't affect the outcome. It's just as easy for salaried programmers to betray their bosses and their users.
2. Stop accepting shit tools like M4 and C for creating any software.
You can never make the work of black hats impossible but you can trivially make it a whole lot harder.
---
> You:
> Below is a hexdump of a test file added to our code repository. Is it any cause for concern?
```
0000000 375 7 z X Z \0 \0 004 346 326 264 F 002 \0 ! 001
0000010 \b \0 \0 \0 330 017 # 023 001 \0 \f # # # # H
0000020 e l l o # # # # \0 \0 \0 \0 022 210 337 004
0000030 Y r 201 B \0 001 % \r q 031 304 266 037 266 363 }
0000040 001 \0 \0 \0 \0 004 Y Z 375 7 z X Z \0 \0 004
0000050 346 326 264 F 002 \0 ! 001 \b \0 \0 \0 330 017 # 023
0000060 340 005 026 001 _ ] \0 005 245 - U 273 230 004 |
0000070 306 231 274 240 f M 315 1 255 \v ) 270 ( \f .
0000080 274 3 263 ~ * 323 312 n 324 C 034 Q Y 314 217 341
0000090 350 343 357 @ 227 271 247 w 026 255 237 U * H 367 240
00000a0 K 260 p 9 5 370 j 017 203 ? 307 ] 005 326 220 240
00000b0 352 I 236 ` ? 304 306 365 246 375 226 [ 257 a 177 -
00000c0 261 033 204 001 203 251 346 252 243 244 < 016 h 256 273 356
00000d0 320 206 ! 355 367 P ) 247 035 226 357 026 224 324 u K
00000e0 255 8 277 326 350 347 205 \n : 220 334 342 u 216 M 357
00000f0 \f = c 316 244 % \0 233 E 354 x " 376 250 266 247
0000100 ^ 303 S 273 220 271 0 | 232 222 252 033 251 \r 372 g
0000110 n 221 177 ) 270 l \v X 315 w , ( \t 254 * $
0000120 023 031 ! | 003 J 0 262 002 030 230 321 4
0000130 340 233 216 337 360 261 \ ] 216 016 020 206 001 357 346 332
0000140 337 z 306 363 374 ] 354 323 335 [ 337 214 z 330 226 027
0000150 j 255 037 260 ' 027 356 333 F W V 200 030 H ; 337
0000160 \n 245 q 7 372 360 I 263 ; 326 316 321 306 030 q K
0000170 337 264 030 1 345 z 032 @ 357 301 ] 302 U < 037 262
0000180 363 272 241 231 346 s \0 337 216 235 375 236 [ 037 215 037
0000190 274 341 033 200 \0 \0 \0 \0 u 216 336 U 336 r 1 u
00001a0 \0 001 311 002 227 \n \0 \0 \n 224 r 242 261 304 g 373
00001b0 002 \0 \0 \0 \0 004 Y Z 375 7 z X Z \0 \0 004
00001c0 346 326 264 F 002 \0 ! 001 \b \0 \0 \0 330 017 # 023
00001d0 001 \0 \r # # # # W o r l d # # # #
00001e0 \n \0 \0 \0 x 360 \v ) 314 g 337 330 \0 001 & 016
00001f0 \b 033 340 004 037 266 363 } 001 \0 \0 \0 \0 004 Y Z
0000200
```
> The hexdump you've provided appears to be a binary file. It's difficult to determine if it's a cause for concern just by looking at the hexdump. Here are a few things you could consider:
> 1. *File Origin*: If this file was added by a trusted team member and it's expected to be part of your code repository, it's probably safe. If the file's origin is unknown or it was added unexpectedly, you might want to investigate further.
> 2. *File Purpose*: Understanding what this file is used for can help determine if it's a cause for concern. If it's a binary used for testing or a necessary part of your application, it might be fine. If its purpose is unclear, it's worth checking.
> 3. *File Analysis*: You could use file analysis tools to examine the binary file. This could provide information about what the file does, which can help you decide if it's a cause for concern.
> 4. *Security Scans*: Consider running security scans on the file. There are many tools available that can scan binary files for known security vulnerabilities.
> 5. *Code Review*: If this file was added as part of a commit, make sure the commit was reviewed thoroughly. Unexpected binary files can sometimes be a sign of an insecure direct object reference (IDOR) vulnerability.
> Remember, when in doubt, it's always better to err on the side of caution. If you're unsure, consider reaching out to a security professional. They can help you analyze the file and determine if it poses any risk.
Lol.
---
> 2. Stop accepting shit tools like M4 and C for creating any software.
> You can never make the work of black hats impossible but you can trivially make it a whole lot harder.
What could be more trivially done than simply replacing C for all code?
Jia Tan, is that you?
https://github.com/tukaani-project/xz/commits?author=thesame...
Seems like he's a Gentoo contributor.
Do we need at least some form of basic identity verification (even in the form of an employer link, LinkedIn, or similar)?
I don't believe that "identity verification" for F/OSS developers is a good idea. Suppliers? Sure. That can be a contract negotiation when you decide how much you pay for it.
Also, I don't think identity verification helps when your adversary is a nation state, which can just falsify government identification if it suits them.
[1] https://scottarc.blog/2024/04/04/open-source-supply-chains-a...
Fair.
> This person/team used a VPN. Masking your location is a big red flag for just dev work like this. These things could be exposed in UI.
I disagree strongly, and am surprised to hear this argument on Hacker News of all places.
volunteer osint researchers attempt to dox any identity you submit, so you can know whether they're the Right Kind of OSS contributor or not.
/s
We shouldn't need to know who you are to be verify that your contribution is trustworthy. Code should stand on its own merits. Ideas like this tend to dovetail into "real name policies", which suck https://www.eff.org/deeplinks/2014/09/facebooks-real-name-po...
You can't solve identity, and real, verified people can still betray you. Even trustworthy people can betray you, either unwillingly or unwittingly.
You can't solve vulnerability analysis either (it's the halting problem), but you also can't shirk it. You might as well go all in on it rather than undermining the basis of open source collaboration and many eyes making bugs shallow.
But it certainly is a wake up call of some kind.
For dedicated attackers (as was the case here) that will move the goal posts ever so slightly farther away, but not much else. I see how it's tempting to focus on identity, but I don't see how it's productive.
Not all free software contributors are employed. They might be too young (it's not unusual to start contributing to free software while still in school), too old and already retired, be part of a family where another family member is the working one, do freelance or informal work, or be searching for work at the moment.
And even those who are employed do not necessarily have it visible on their employer's website; for instance, I don't think my current employer has any publicly available page listing all employees, and I never had (and don't have any desire for) a LinkedIn account.
Obviously, every system is going to have weaknesses, but it would at least introduce some accountability.
The main issues are all related to privacy. What if your government is not nice and you don't want them to track all your contributions?
Additionally, creating fake identities in other states is base level spying.
Maybe not. But to the open source community it does provide more information. Now it's completely unclear who compromised xz, if the account had a government-level signature of some sort, more is known. Also, the open source community could have different levels of trust in authors with different signing authorities [1].
Additionally, creating fake identities in other states is base level spying.
We are not talking about a fake identity here (as in a fake passport or whatever), but getting a signature from a device that is presumably in a facility that is disconnected from the network. The user's identities would also live on a hardware device (e.g. chip embedded in a passport/id).
You could steal the hardware device to steal someone's developer identity. But the identity can be revoked once the device is reported/stolen. So, it is hard to play the long game with a stolen identity, outside blackmailing or otherwise compelling a developer to make their identity available (but even in that case it'd give some leads in understanding who the actor is).
Apple Developer signing keys are similar. You can verify where an app comes from. If a developer goes rogue or their key is compromised, Apple can revoke it. Apple could do evil stuff by creating sock puppet accounts, but that would devalue the trust in their signing key.
---
[1] I should note that I am not in favor of this. It's a privacy nightmare and it would give governments a lot of control over developers. I just think the argument upthread that a state actor could fake anything is not true. If e.g. the US had such an identity signing system, a foreign power could not forge the identities as long as the US private key is not compromised.
Take your pick! 1600+ to choose from.
This seems false? The trie contained also strings like `WAYLAND_DISPLAY=`, so IIUC the backdoor would remain dormant if someone tried to run sshd from something that looks like an interactive session.
The various Jia Tan sockpuppets were actively pushing both Debian and Redhat to include the backdoored version of xz, so the answer to your last question is yes, they are in fact active participants in the community.
I think there is a small de-blobbed kernel but it is unpopular
It doesn't matter anyway, because once you're running inside a process you can do most anything you like. This was a convenient mechanism, but there are scores of ways to scan over the symbols and modify code inside the same process.
Now, this is why we can't have secure systems.
Somehow, I've never needed to attach GDB to a running process. Start up programs under GDB, yes. Read crash dumps, yes. If you're using a debugger in a production system, something has gone horribly wrong.
That's right, hence the debugger.
But yes, some black hat had a really really bad day: being so close to owning any ssh-enabled recently-updated box and when they can finally see a finish line it is just poof... gone in a day. It was two years of effort from at least one guy and more likely a team.
Aside, this is one of my grumpy-old-dude opinions when it comes to regular non-malicious code: Source "greppability" is a virtue, one that must be balanced against "don't repeat yourself" and "compile time checking", etc.
Some examples off the top of my head:
1. One import/alias per line, no wildcards.
2. If you have to concatenate a literal string across multiple lines, try to ensure the break isn't inside an interesting substring someone would search for.
3. If you write a compile-time constant like foo=4*6*9, the comments around it should also contain the result. Sure, someone might screw up and forget to keep the comment in-sync, but it can also be invaluable when someone else is tearing their hair out trying to find out why the production logs are saying nothing but "Illegal frump: 216".
Don't get me wrong, I love a good IDE, and many will offer "Find Usages" or "Find Literal"... but it doesn't always work all the time, for all languages in the project at once, for everyone on a team who might use other tools, or during a PR review through a web-browser, etc.
This is called semantic line breaks and is also important to get (admittedly more naive) diffs sensible.
The fact that many projects are now searching deep and wide through their commit histories, looking at just who they're taking code from, beginning to develop frameworks for attack mitigation and remediation... an entire type of previously very-promising attack is completely burned. This has been a massive defeat. And it happened all by chance.
I fear it's not just the attack that is burned. If new contributors have to be distrusted and/or go through some sort of vetting that isn't based on the merit of their contributions, that is a terrible blow to the entire open source movement.
The threshold for young coders without a great deal of history and without a network of contacts to become contributors to important open source projects has just gone up massively.
The bar is still trivial. The worst part is that it will be difficult to be anonymous. There are a lot of valid, even life & death reasons for anonymity, and that will now be much harder.
So the loss is all the contributors who don't dare let their government or employer see them helping the wrong projects, or in some cases don't want to disclose that they are any good at coding or particularly interested in security etc.
I would expect to see more projects try to protect against this in general by attempting to separate the building and testing procedures.
No, it was not.
The test files were used as carriers for the bulk of the malicious code, but running the tests did nothing malicious; the supposedly corrupted test file did act as a corrupted file in the test. What extracted and executed the code hidden in these test files was a bit of code injected in the configuration steps, which in turn modified the compilation steps to extract and execute more code hidden in these test files, which in turn extracted and injected the backdoor code. This all happened even if the tests were never run.
> I would expect to see more projects try to protect against this in general by attempting to separate the building and testing procedures.
The suggestions I've seen on this line of thinking were to not only separate the building and testing steps (so that the testing steps cannot affect the output of the build step), but also to remove all test files before doing the build, and adding them back for the testing step. The important part being to remove all binary test files before doing the build, and having two separate steps only as a side effect of not being able to test without putting the test files back.
I don't think this approach is going to be adopted, as it's too much work for little gain; people are focusing more on making sure all files on the tarball used for the build either come from the corresponding tag in the repository, or can be proven to have been deterministically generated from files found in the corresponding tag in the repository. That would have either caught or ignored the key file added to the tarball which started the extraction and injection steps.
Just make the two separate projects, where the test project depends on installed binaries.
Assuming they work 40 hours a week and are doing this in a team (presumably every major player has such a team or is scrambling to get one now), one must expect many potentially infiltrated projects out there.
The government who paid them just lost the investment, but who cares about them? It's only a good thing if this does not get a reputation for being a good invesment.
If it was a criminal, then the same as the government they lost the investment but again that's only a good thing.
> If so, then they were paid for those hours and are perfectly whole right now.
People are not ideal money-fed automatons. Even if you're fully paid for your software development, it feels bad to see all your work thrown away.
> maybe their career takes a dent but I do not weep for anyone who worked on this
Even if they were fully paid, and even if their career is not affected at all (or even improves, "I wrote the xz backdoor" would be an impressive line in a curriculum if you disregard the moral aspects), it can still feel bad to see your work thrown away so close to the finish line. People are not career-driven automatons.
But I agree with you, I do not feel bad for the author of this backdoor; whoever did that does deserve to see this work unceremoniously dumped into the trash. But I can understand why the ones who worked on that code would feel bad about it.
I don't see why; news coverage has pretty uniformly taken the view that "this guy was an evil genius, and we escaped his incredibly well-executed plot by the skin of our teeth".
The statement was that someone had a really bad day.
The implication that people are driven by money and nothing else, or even that they have no right to feel like they had a bad day if they were paid, is absurd.
Nobody is saying that you should be sympathetic. It’s just an interesting comment: An interesting thing to think about. A worthwhile contribution to the conversation. This was someone’s very bad ending to a very long project.
You think these guys have standups?
I'm not familiar with any programming language that provides tries in a standardized way. They're not so hard to code, so I wonder if this will become a trend in future malware.
The honorable thing for "Jia" to do after this epic failure is seppuku, or whatever his or her local equivalent is.
People like "Jia" are pure parasites. It's one of the best cases of "why we can't have nice things" I've ever seen.
It would definitely be interesting to see what would happen if the attack wasn't noticed, but instead people focus their interest on attacking Jia Tan because "wow, that guy is one hell of an asshole, it sure is a great thing that he failed!".
Whether or not this attack was the rare one that failed out of many similar ones is largely irrelevant to people. Quick, discuss this one particular case where we noticed it, news flash and all.
> People like "Jia" are pure parasites
They are "parasites" because they don't do what they are "supposed to"? That's pretty crazy. I guess every person that's doing what matches their interests is somehow a bad person/parasite/etc. Or is that only if what they do is forbidden by some rule book? Do you see what I'm getting at here?
"pls sir dont create linux backdoar"
If they inflict harm on their fellow humans in order to do so, then literally yes.
There's a reason why a society will collapse if you have too many of this type of person.
What I'm saying is that there are a lot more technically legal ways to profit that harm society, some of them more nefarious than what Jia Tan did.
Doing things that are bad for the society in a fucked up society seems justifiable. It doesn't necessarily make you a bad person.
People just have a more averse reaction to things that are obviously bad, even if in practice there are way worse things that initially seem innocuous and are actually legal to do. That's just the textbook example of hypocrisy.
I think you're missing where the difficulty is. I'd argue the technical knowledge here isn't the hard part here. You can find more than enough folks who would know how to do everything you listed outside of any state-sponsored context. The hard part is the methodical execution, including the opsec, backup plans, plausible deniability, false identities, knowing how to play the long game, being able to precisely target what you want, understanding human psychological tendencies, identifying unforeseen security vulnerabilities, etc... these are the things that I imagine are likely to distinguish talented individuals from state-sponsored actors, not something like "knows tries and x86 assembly".
And taking all those into account, this absolutely smells state-sponsored. I just see no way that a private entity had enough of an incentive, foresight, experience, and patience to play such a long game with such careful maneuvers.
The assault was comprehensive, holistic and systematic in its approach – this article does not mention it, but other reports have indicated that the person behind it also managed to compromise the PKI layer at the edge between OpenSSL and sshd which brings an extra level of complexity to the backdoor.
Doesn't this apply to many OSS contributors? people with skills and free time?
https://rheaeve.substack.com/p/xz-backdoor-times-damned-time...
There are other countries on the same time zone, but only Israel has history of using and selling hacks of this level.
Or someone further east with a classical hacker "let's get up late in the afternoon, and start the serious hacking late at night."
Criminals with the resources to maintain such a complex operation for two years?
I also am a little biased and assume being burned in state-sponsored acts is similar to the no-blame culture of breaking infrastructure in tech :) because by all accounts this compromise was extremely well done, until it wasn't.
Also, we can't be sure the compromise wasn't intentionally telegraphed to cause some other action (using a different library) on purpose.
That actually makes me think it's not happening at a larger scale, since we'd likely have heard of at least a few similarly elaborate cases being uncovered by now. If not during the attempt itself, then at least at some later point in time.
Either almost all of these operations remain undetected, because they are even more sophisticated and much of the world's software ecosystem has been secretly compromised for years or there aren't actually that many such operations.
It's one thing to attack a target, it's quite another to try to give yourself a master key to every rpm or deb linux box in the world.
Quite confusing, because the two different meanings are nearly opposite to one another.
See also: https://www.merriam-webster.com/grammar/nonplussed
It's pretty easy to see where the innovative sense came from: "plussed" doesn't mean anything, but "non" is clearly negative. So when you encounter the word, you can tell that it describes (a) a reaction in which (b) something doesn't happen. So everyone independently guesses that it means failing to have much of a reaction, and when everyone thinks a word means something, then it does mean that thing.
You see the same thing happen with "inflammable", where everyone is aware that "in" means "not" and "flame" means "fire". (Except that in the original sense, in is an intensifying prefix rather than a negative prefix. This doesn't occur in many other English words, although "inflammation" and "inflamed" aren't rare. Maybe "infatuate".)
Once the first stage, extracting a shell script from one of the „test“ data blobs, has been found it was clear to everybody that something fishy is going on.
It’s inconceivable that I’ve would have found the first stage and just given up, but then it was „only“ a matter of tedious shell reversing…
They could easily done without the „striping“ or „awk RC4“, but that must have complicated their internal testing and development quite a bit.
But what you were looking at might not be the first stage.
You might be looking at the modified Makefile. You might be looking at the object files generated during the build. You might be looking at the build logs. You might be investigating a linking failure. The reason for so many layers of obfuscation, is that the attacker had no idea at which layer the good guys would start looking; at each point, they tried to hide in the noise of the corresponding build system step.
In the end, this was caught not at the build steps, but at the runtime injection steps; in a bit of poetic justice, all this obfuscation work caused so much slowdown that the obfuscation itself made it more visible. As tvtropes would say, this was a "Revealing Cover-Up" (https://tvtropes.org/pmwiki/pmwiki.php/Main/RevealingCoverup) (warning: tvtropes can be addictive)
For a one of its kind deployment it would probably not matter. However, deploying to multiple targets using the same basic approach would allow all of them to be found once one was discovered. With some mildly confsing but different scripting for each target systematic detection of others becomes more difficult.
Because anyone has the resources to test this locally and find the issues, but they just didn't have the time anymore.
Or was xz only one of multiple targets where the overall plan didn't actually fail and an updated backdoor, without the performance issues, will soon be injected again through some other project.
Surely that would be something the current ais could do 100x better?
So I wonder if there isn't a business there in scanning anonymous posts to associate them with others to deanonymize them all?
You could obviously use an ai to rewrite your text, but I bet there is something there that can still be correlated. Simply avoiding using favorite words and phrases doesn't change the essence of whatever you're saying.
I think the concept of sockpuppet is about to get complicated. Really it probably already is. If I can imagine something, someone else has probably already been doing it and countering it, and countering that for a decade.
https://theconversation.com/how-the-unabombers-unique-lingui....
Jia Tan mook: "Sir, they found us out."
Jia Tan officer: "What's the sitrep, are they coming for us? Are they getting our operatives in the field?"
Jia Tan mook: "No sir, they're just reverting and blacklisting our accounts and releases. Muttering something about 'uptime'."
Jia Tan officer: "Shit, we're dealing with sysadmins."
and if 2 is true, the KGB university alumni Kaspersky definitely is related to it.
https://en.wikipedia.org/wiki/Kaspersky_bans_and_allegations...
Which is true, and the number of actual running systems that the backdoor already made its way into is likely not massive -- especially in terms of production or otherwise critical systems.
But that doesn't mean it wasn't bad. It's rather a disaster that was only avoided pretty much by accident.
Public networks have too many eyes potentially (not just the admins responsible for specific systems). Traffic to ssh server that corresponds to suspicious activity but is also absent from logs would raise the alarms quite soon.
Hooking the build process also means that the main targets are places that are important enough to have their own siloed software repositories. Getting into two LTS systems was a bonus. I suppose that simpler exploits are available for popular systems (if they run regular complex applications instead of being a minimal empty box that securely does nothing).
The number of modification/decryption layers is probably not arbitrary. Most likely, each was added to evade detection by one or another protection system you and I don't know about, but with which attacker is familiar.
Junk packets to port 22 arrive all the time.
I suppose “smart” readers can be as easily manipulated to look in any wrong direction as “dumb” ones. All stories mention what each decryption and indirection step does, but they don't explain why exactly those steps were chosen, that this one hides from intrusion detection system A, and that one prevents collection of runtime activity artifacts by tracing system B, and so on. On one hand, it would be impolite to announce that you study, and know well, how to break your colleague's product, on the other hand, some of those systems might be secret if we talk about state level.
All this adds up to a group which wanted to have an easy way into a relatively small but diverse set of high-value targets, it's likely they would have waited for the targeted distro released to reach fairly widespread adoption before exploiting it, and they would have tried to keep it low-profile enough that it would not be discovered for some time.