Code review? Some kind of behavioral analysis?
IMO the call to system() was kind of sloppy, and a binary capabilities scanner could have potentially identified a path to that.
Games and proprietary software will sometimes ship with DRM protection layers that do insane things in the name of obfuscation, making it hard to distinguish from malware.
But (with only a couple exceptions) there's no reason for a binary or library in a Debian package to ever try to write the PLT outside of the normal mechanism, to try to overwrite symbols in other modules, to add LD audit hooks on startup, to try to resolve things manually by walking ELF structures, to do anti-debug tricks, or just to have any kind of obfuscation or packing that free software packaged for a distro is not supposed to have.
Some of these may be (much) more difficult to detect than others, some might not be realistic. But there are several plausible different ways a scanner could have detected something weird going on in memory during ssh startup.
No one wants a Linux antivirus. But I think everyone would benefit from throwing all the behavioral analysis we can come up with at new Debian package uploads. We're very lucky someone noticed this one, we may not have the same luck next time.
People still manage to write exploits today, but now you must find an ASLR leak, you must chain enough primitives to work around multiple layers of protection, it's generally a huge pain to write exploits compared to the 90s.
Today the dynamic detection that we have for Linux packages seems thin to non-existent, like the arms race has not even started yet. I think there is a bit of low-hanging fruit to make attacker lives harder (and some much higher-hanging fruit that would be a real headache).
Luckily there is an asymmetry in favor of the defenders (for once). If we create a scanner, we do not _have_ to publish every type of scan it knows how to do. Much like companies fighting spammers and fraud don't detail exactly how they catch bad actors. (Or, for another example, I know the Tor project has a similar asymmetry to detect bad relays. They collaborate on their relay scanner internally, but no one externally knows all the details.)
Indeed, defenders are often outgunned, but at a minimum we’ve pushed the level of effort way into nation-state territory, which is a good start.
I want to name one thing: when Windows failed to load a DLL because a dependency was missing, it doesn't tell you what was missed. To get the information, you have to interact with the DLL loader with low level Windows APIs. In some circumstances Linux apps may also have the need. Like for printing a user friendly error message or recovery from a non-fatal error. For example, the patchelf tool that is used for building portable python packages.
> No one wants a Linux antivirus
It is not true. Actually these software are very popular in enterprise settings.
ClamAV has been around for a very long time at this point.
It's just not installed on servers, usually
A cloud provider can take snapshots of running VMs then run antivirus scan offline to minimize the impact to the customers.
Similarly, many applications are containerized and the containers are stateless, we can scan the docker images instead. This approach has been quite mature.
If the prior is 1 was out there (this one), the chances that there is 1+ still undetected seems fairly high to me.
To behaviourally detect this requires many independent actors to be looking in independent ways(e.g. security researchers, internal teams). Edit: I mean with private code & tests (not open source, nor purchasable antivirus). It's not easy to donate to Google Zero. Some of the best funded and most skilled teams seem to be antivirus vendors (and high value person protection). I hate the antivirus industry yet I've been helped by it (the anti-tragedy of the commons).
Commonly public detection code (e.g. open source) is likely to be defeated by attackers with a lot of resources.
Hard to protect ourselves against countries where the individuals are safe from prosecution. Even nefarious means like assasination likely only work against individuals and not teams.
That would surprise me greatly
Why "new uploads" and not also "all existing"?
Now, if there was a proper chain of accountability for data, then this wouldn't have been possible to hide the way it is - any amount of pre-processing resulting in the release tarball including derived products of "test" files would be suspicious.
The problem is we don't actually track data providence like this - no build system does. The most we do is <git hash in> -> <some deterministic bits out>. But we don't include the human readable data which explains how that transform happens at enough levels.
It’s not sufficient against a determined attacker, but it does demonstrate just how unserious the C world is about their build engineering.
I literally can’t think of a single time in 15 years of work that I’ve ever seen a reason for a dist build to need test resources. That’s at best a bug - if it’s a dist resource it goes in the dist resources, not test. And if the tooling doesn’t do a good job of making that mistake difficult… it’s bad tooling.
A bytecode interpreter that can call syscalls can be just a few hundred bytes of code, and means you can avoid calling system() (whose calls might be logged), and avoid calling mprotect to make code executable (also something likely to raise security red flags).
The only downside of a bytecode interpreter is the whole of the rest of your malware needs to be compiled to your custom bytecode to get the benefits, and you will take a pretty big performance hit. Unless you're streaming the users webcam, that probably isn't an issue tho.
As for the rest of us, we got lucky. In fact, it’s quite hilarious that some grump who’s thanklessly perf testing other people’s code is like “no like, exploit makes my system slower”.
Andres is one of the most prolific PostgreSQL committers and his depth of understanding of systems performance is second to none. I wouldn't have guessed he would one day save the world with it, but there you go.
Had sshd been written in go or rust, the attack would have still been possible.
If xz was statically linked in some way, or just used as an executa Le to compress something (like the kernel), the same problems exist and no dynamic linking would need to be involved.
[1] https://sourceware.org/glibc/wiki/GNU_IFUNC
Dynamic loading is relic of the past and cause of many headaches in linux ecosystem, in this case it also just obfuscates the execution path of the code more so you can't really rely on the code you are reading. Unfortunately I don't think it's possible to completely get rid of dynamic loading as some components such as GPU drivers require it, but it should be reduced to minimum.
The point is, there's simply no usecase to require or even allow the program to do IFUNC substitution freely on its own. A programming framework should not opt the developer in to capabilities they don't want or need. Much of C-likes' complexity arises from unnecessary, mandated capabilities.
It seems foolish to change our systems to stop one of the steps the attacker used after their code was already running as root; the attacker can just pick something else; as root they have essentially unlimited options.
Rejecting evidence in favor of retaining preconceptions is certainly popular.
On the other hand, dynamic linking was originally more or less just a hack to deal with memory-restricted environments in the face of growing amounts of code. It was necessary at the time because we simply wouldn't have things like X or Windows without it way back when.
But RAM is nowhere near as sparse these days, and it could be even less so if there was a concerted push on hardware vendors to stop skimping on it. So why don't we remove the hack and get back to a simple model that is much easier to understand, implement, and audit?
even more so: all binaries dynamically linking xz can be updated by installing a fixed library version. For statically linked binaries: not so much, each individual binary would have to be relinked, good luck with that.
Is essentially what all distros with sane build systems does.