Keeping veil of secrecy requires unimaginable amount of energy. Same goes with truth consistency. One little slip and everything goes to nothing. Sometimes single sentence can start a chain of reaction and uncover meticulous crafted plan.
That’s how crime if fought every day. Whereas police work has limited resources, software is analyzed daily by hobbyists as a hobby, professionals who still do it for a hobby, and professionals for professional reasons.
Discovery was bound to happen eventually.
XZ attack was very well executed. It’s a master piece. I wouldn’t be surprised if some state agencies would be involved. But it also was incredibly lucky. I know for sure for myself, but also many of my colleagues would go into long journey if found any of issues that are flagged right now.
One takeaway is that chance of finding such issue would be impossible if xz/liblzma wouldn’t be open source (and yes I am also aware it enabled it in the first place) but imagine this existing in Windows or MacOS.
it took roughly two years including social engineering.
I'd say the same approach is much easier in a big software company.
In a big company it's much easier to slip it in. Code seemingly less relevant for security is often not reviewed by a lot of people. Also, often people don't really care and just sign it off without a closer look.
And when it's merged, no one will ever look at it again, other than with FOSS.
That said, if you're not using it, it defeats the purpose. And the more you're using it, the higher the likelihood you will be detected down the line. Compare to Solarwinds.
Most people don't log TCP connections, and those that do don't go through their logs looking for odd certificates in ssh connections.
And no common logging at the ssh/pam level would have picked this up.
Your only chance is some sysadmin who has put 'tripwires' on certain syscalls like system(), fork() or mmap() looking for anything unusual.
Even then, they might detect the attack, yet have no chance at actually finding how the malicious code loaded itself.
Even then, who actually has a massive fleet of publicly exposed servers all running a mix of distros/versions? You might run a small handful of distros, but I suspect anyone running a fleet large enough to actually collect a substantial amount of data probably also has tools to upgrade the whole fleet (or at least large swaths) in one go. Certainly there are companies where updates are the wild west, but the odds that they're all accessible to and controllable by a single motivated individual who can detect the exploit is essentially zero.
I’m not arguing with the point, but this is a great place to hide — very difficult to have meaningful detection rules even for a sophisticated sysadmin.
I _think_ it’ll look very different in ps —-forest output.
It’s certainly possible, but my guess is sshd is likely to have a lot of open policy. I’m really curious if someone knows different and there are hard detection for those things. (Either way, I bet there will be in the future!)
Not always. Weapons of war are most useful when you don't have to actually use them, because others know that you have it. This exploit could be used sparingly to boost a reputation of a state-level actor. Of course, other parties wouldn't know about this particular exploit, but they would see your cyber capabilities in the rare occasions where you decided to use it.
Except that cyber weapons like these are
1. One time use 2. Expire upon detection (mostly)
I think this is simply just a tool for offensive action only.
Not if this was injected by a state actor. My experience with other examples of state actor interference in critical infrastructure, is that the exploit is not used. It’s there as a capability to be leveraged only in the context of military action.
Why do non-friendly state actors (apparently) not detect and eliminate exploits like this one?
Supposedly, they should have the same kind of budgets for code review (or even more, if we combine all budgets of all non-friendly state actors, given the fact that we are talking about open-source code).
When a state actor says "We found this exploit", people will get paranoid and wondering if the fix is actually an exploit.
Not saying it happened in this case, but it's really easy for a state actor to hide an extensive audit behind some parallel construction. Just create a cover story pretending to be a random user who randomly noticed ssh logins being slow, and use that story to point maintainers to the problem, without triggering anyone's paranoia, or giving other state actors evidence of your auditing capabilities.
They wouldn't be able to do that for this particular exploit since it requires successfully decrypting data encrypted by the attacker's secret key. A zero day caused by an accidental bug though? There's no reason for them to eliminate the threat by disclosing it. They can patch their own systems and add yet another exploit to their hoard.
"Their own systems" will necessarily include lots of civilian infrastructure. Hard to make sure all that gets patched without issuing a CVE, let alone without anyone in the general public even being aware of the patch.
After something like 2 years of planning this out and targeted changes this isn't something "just done for science".
And what's about the fellows of U of Minnesota?
The University of Minnesota case is an interesting one to compare to. I could imagine them being criminally liable but being given a lenient punishment. I wonder if the law will end up being amended to better cover this, if it isn’t already explicitly illegal.
Well, software supply chains are a thing.
"where no auditor ever is paid to look" would be more correct.
Code review? Some kind of behavioral analysis?
IMO the call to system() was kind of sloppy, and a binary capabilities scanner could have potentially identified a path to that.
Games and proprietary software will sometimes ship with DRM protection layers that do insane things in the name of obfuscation, making it hard to distinguish from malware.
But (with only a couple exceptions) there's no reason for a binary or library in a Debian package to ever try to write the PLT outside of the normal mechanism, to try to overwrite symbols in other modules, to add LD audit hooks on startup, to try to resolve things manually by walking ELF structures, to do anti-debug tricks, or just to have any kind of obfuscation or packing that free software packaged for a distro is not supposed to have.
Some of these may be (much) more difficult to detect than others, some might not be realistic. But there are several plausible different ways a scanner could have detected something weird going on in memory during ssh startup.
No one wants a Linux antivirus. But I think everyone would benefit from throwing all the behavioral analysis we can come up with at new Debian package uploads. We're very lucky someone noticed this one, we may not have the same luck next time.
People still manage to write exploits today, but now you must find an ASLR leak, you must chain enough primitives to work around multiple layers of protection, it's generally a huge pain to write exploits compared to the 90s.
Today the dynamic detection that we have for Linux packages seems thin to non-existent, like the arms race has not even started yet. I think there is a bit of low-hanging fruit to make attacker lives harder (and some much higher-hanging fruit that would be a real headache).
Luckily there is an asymmetry in favor of the defenders (for once). If we create a scanner, we do not _have_ to publish every type of scan it knows how to do. Much like companies fighting spammers and fraud don't detail exactly how they catch bad actors. (Or, for another example, I know the Tor project has a similar asymmetry to detect bad relays. They collaborate on their relay scanner internally, but no one externally knows all the details.)
Indeed, defenders are often outgunned, but at a minimum we’ve pushed the level of effort way into nation-state territory, which is a good start.
I want to name one thing: when Windows failed to load a DLL because a dependency was missing, it doesn't tell you what was missed. To get the information, you have to interact with the DLL loader with low level Windows APIs. In some circumstances Linux apps may also have the need. Like for printing a user friendly error message or recovery from a non-fatal error. For example, the patchelf tool that is used for building portable python packages.
> No one wants a Linux antivirus
It is not true. Actually these software are very popular in enterprise settings.
ClamAV has been around for a very long time at this point.
It's just not installed on servers, usually
A cloud provider can take snapshots of running VMs then run antivirus scan offline to minimize the impact to the customers.
Similarly, many applications are containerized and the containers are stateless, we can scan the docker images instead. This approach has been quite mature.
If the prior is 1 was out there (this one), the chances that there is 1+ still undetected seems fairly high to me.
To behaviourally detect this requires many independent actors to be looking in independent ways(e.g. security researchers, internal teams). Edit: I mean with private code & tests (not open source, nor purchasable antivirus). It's not easy to donate to Google Zero. Some of the best funded and most skilled teams seem to be antivirus vendors (and high value person protection). I hate the antivirus industry yet I've been helped by it (the anti-tragedy of the commons).
Commonly public detection code (e.g. open source) is likely to be defeated by attackers with a lot of resources.
Hard to protect ourselves against countries where the individuals are safe from prosecution. Even nefarious means like assasination likely only work against individuals and not teams.
That would surprise me greatly
Why "new uploads" and not also "all existing"?
As for the rest of us, we got lucky. In fact, it’s quite hilarious that some grump who’s thanklessly perf testing other people’s code is like “no like, exploit makes my system slower”.
Andres is one of the most prolific PostgreSQL committers and his depth of understanding of systems performance is second to none. I wouldn't have guessed he would one day save the world with it, but there you go.
A bytecode interpreter that can call syscalls can be just a few hundred bytes of code, and means you can avoid calling system() (whose calls might be logged), and avoid calling mprotect to make code executable (also something likely to raise security red flags).
The only downside of a bytecode interpreter is the whole of the rest of your malware needs to be compiled to your custom bytecode to get the benefits, and you will take a pretty big performance hit. Unless you're streaming the users webcam, that probably isn't an issue tho.
Now, if there was a proper chain of accountability for data, then this wouldn't have been possible to hide the way it is - any amount of pre-processing resulting in the release tarball including derived products of "test" files would be suspicious.
The problem is we don't actually track data providence like this - no build system does. The most we do is <git hash in> -> <some deterministic bits out>. But we don't include the human readable data which explains how that transform happens at enough levels.
It’s not sufficient against a determined attacker, but it does demonstrate just how unserious the C world is about their build engineering.
I literally can’t think of a single time in 15 years of work that I’ve ever seen a reason for a dist build to need test resources. That’s at best a bug - if it’s a dist resource it goes in the dist resources, not test. And if the tooling doesn’t do a good job of making that mistake difficult… it’s bad tooling.
Had sshd been written in go or rust, the attack would have still been possible.
If xz was statically linked in some way, or just used as an executa Le to compress something (like the kernel), the same problems exist and no dynamic linking would need to be involved.
[1] https://sourceware.org/glibc/wiki/GNU_IFUNC
Dynamic loading is relic of the past and cause of many headaches in linux ecosystem, in this case it also just obfuscates the execution path of the code more so you can't really rely on the code you are reading. Unfortunately I don't think it's possible to completely get rid of dynamic loading as some components such as GPU drivers require it, but it should be reduced to minimum.
The point is, there's simply no usecase to require or even allow the program to do IFUNC substitution freely on its own. A programming framework should not opt the developer in to capabilities they don't want or need. Much of C-likes' complexity arises from unnecessary, mandated capabilities.
It seems foolish to change our systems to stop one of the steps the attacker used after their code was already running as root; the attacker can just pick something else; as root they have essentially unlimited options.
Rejecting evidence in favor of retaining preconceptions is certainly popular.
On the other hand, dynamic linking was originally more or less just a hack to deal with memory-restricted environments in the face of growing amounts of code. It was necessary at the time because we simply wouldn't have things like X or Windows without it way back when.
But RAM is nowhere near as sparse these days, and it could be even less so if there was a concerted push on hardware vendors to stop skimping on it. So why don't we remove the hack and get back to a simple model that is much easier to understand, implement, and audit?
even more so: all binaries dynamically linking xz can be updated by installing a fixed library version. For statically linked binaries: not so much, each individual binary would have to be relinked, good luck with that.
Is essentially what all distros with sane build systems does.
As for being discovered outside the target, well we tried that exercise already, didn’t we? A bunch of people stared at the payload with valgrind et al and didn’t see it. It’s also fairly well protected from being discovered in debugging environments, because the overt infrastructure underlying the payload is incompatible with ASan and friends. And even if it is linked in, the code runs long before main(), so even if you were prodding around near or in liblzma with a debugger you wouldn’t normally observe it execute.
e: sibling suggests strace, yes you can see all syscalls after the process is spawned and you can watch the linker work. But from what I’ve gathered the payload isn’t making any syscalls at that stage to determine whether to activate, it’s just looking at argv and environ etc.
For something much more heavy-handed, force the pages in sensitive sections to fault, either in the kernel or in a hypervisor. Then look at where the access is coming from in the page fault handler.
I don't think you can reliably differentiate a backdoor executing a command, and a legitimate user logged in with ssh running a command once the backdoor is already installed. But the way backdoors install themselves is where they really break the rules.
Huh, ssh executes things that aren't shells all the time during normal operation. No? i.e. 'ssh myserver.lan cat /etc/fstab'
[0] https://en.wikipedia.org/wiki/PWB/UNIX
[1] https://news.ycombinator.com/item?id=38020792Although initial detection via eg strace would be possible, if the backdoor was later removed or went quiescentit would be full trusting trust territory.
In any non trivial build system, there's going to be lots of third party things involved. Especially when you include tests in the build. Is Python invoked somewhere along the build chain? That's like a dozen libraries loaded already.
Nothing is gained from protecting against an exact replica of this attack, but from this family of attacks.
But seriously, we could have found ourselves with this in all stable repos: RHEL, Debian, Ubuntu, IoT devices 5 years from now and it would have been a much larger shit show.
But one possible reason is if the FBI is convinced that something the NSA is doing is illegal. They may not always be inclined to tolerate that.
I'd noticed that; this seems to have been the case for a long time. You'd think that having state security agencies at war with one-another would be a disaster, but perhaps it's a feature: a sort of social "layered security". At any rate, it seems much better than having a bunch of state security agencies that all sing from the same songsheet.
So rulers in all ages tended to create multiple different security apparatuses for themselves and their states, and often actively encouraged rivalries between them, even if that makes them less efficient.
If you had a human-asset with decision making authority and trust in place, then as funded organization with regular working hours, you'd simply can the project and start prototyping new potential uses.
But I don't think valuable government targets are in any hurry to upgrade. I wouldn't expect widespread adoption of 24.04, even in the private sector, until well after the U.S. election.
By the next election, though, everyone will be running it.
Edit: According to another comment [1], there would only have been a short window of vulnerability during which this attack would have worked, due to changes in systemd. This might have increased pressure on the attacker to act quickly.
Think about backdoors that are already present and will never be found out.
Whats more likely - that this is the only backdoor like this in linux, or that there are more out there and this is the one we happened to find?
I really hope someone is out there testing for all of this stuff in linux:
- Look for system() calls in compiled binaries and check all of them
- Look for uses of IFUNC - specifically when a library uses IFUNC to replace other functions in the resulting executable
- Make a list of all the binaries / libraries which don't landlock. Grep the sourcecode of all those projects and make sure none of them expect to be using landlock.
The attacker would then be stuck inside the jump host and would have to probe where to connect next. This hopefully would then trigger an alert, causing some suspicion.
A shared instance would allow the attacker to just wait for another connection and then follow its traces, without risking triggering an alert by probing.
The ideal jump host would allow to freeze the running ssh process on an alert, either with a snapshot (VM based) or checkpointing (container based), so it can be analyzed later.