XZ was just the exploit we found
How do we protect ourselves from the ones we haven't found yet? What are good security practices that might have lead to this exploit never making it to a dist package.
It’s quite possible it wasn’t meant to stick around longer than a specific operation/breach. For example, it doesn’t run on ARM. It was “always on.” The committer added binary files to the repo (this one… I get the social engineering here… but I can’t see allowing it like this, even for testing a compression lib).
Also, it’s an expensive exploit to deploy. You can’t use this vector anymore.
Now. If someone expected a universal backdoor that would last years, like for law enforcement or intelligence, those things probably wouldn’t work.
On the other hand, if you needed to open a specific attack surface temporarily, it’s pretty brilliant.
Counter-argument is that you’d have to sort of know your target’s upgrade cadence, that the target entry is on x86 (likely, I guess), and that you wouldn’t get caught probing the ssh auth until it works.
So with that, you can go a long way by not exposing anything at all to the internet. If you want to host private services use a VPN or zt mesh like nebula.
That won't solve everything, but it's a start.
but, this is an important question. my personal takeaways/suggestions so far are:
- i question the footprint and other choices like distro-maintained patches that lead systemd to be linked with upstream ssh. this is bad because while we presumably know systemd is critical and we presumably know ssh is critical, the SMEs of either would know little about the other and would not probably expect this at all. nor would the not-SMEs who made the distro-patches. what this really necessary?
- autoconf has always been pretty awful. i also question the need for, or tradition perhaps, that dpkg and rpm use tarballs to build from. would it not be easier to build from a git repo, or even a tar of one? this isn't perfect, but it would make it a lot harder to hide build steps and to ensure that everybody followed the same build steps with the same inputs. reproducible builds might have also helped make it harder on the attacker.
- run what you test and test what you run. the attacker went to some effort to misdirect, but it should have raised flags that ifunc builds couldn't be tested. perhaps we should have said "then we don't need ifunc enabled builds"? oh well, lzma is "just" a compression tool, it's not security critical...
- the entire concept of binary built distros may have reached their zenith. they place a lot of trust in the distributor both to be honest and not make any mistakes. it's not just "don't let the attacker upload rogue binarys (and hashes)", but now also "don't accept/create dishonest or foolish (systemd) patches". reproducible builds from source - sources very much closer to upstream - would help. i've not used Gentoo or Arch, but slackware, LFS, buildroot, and Yocto i have.
- after years of being a doubter of "defense-in-depth", this is one of the first times i see it as important: many security conscious orgs may not have ssh open to the internet. that may well have saved them, had this been even more real.
- overall, the complexity of systemd, linux in general, to a lesser extent, openssh and openssl, to be disappointing. we keep paying a high price for this insane complexity, but i get very close to nothing from it. the compiler does work a bit better now. X (or wayland) still sucks and crashes a lot. pretty much everything else is not better than 10 or 20 years ago. just faster hardware and way more bloated software. why can't we have a dropbear ssh server that does exactly and only what it does best spawned by a simple init? and why does openssh carry around 30 years of baggage?
- in the rush to cure security ills, real and imagined, we've developed "patchitis". but newer isn't always better. this is a perfect example where 5.4 was working 100% fine. upgrading isn't always safe or free. yet many people want to stay on rolling releases and latest builds. especially automatically solving and grabbing the latest version of dependencies, as some tools like npm and pip encourage, has always scared me. upgrading a dependency should be taken more seriously than "NMU update request? well, OK then."
- the secure parts of our system should be simple. we should be able to accurately identify them and where all their code comes from (e.g. not a downstream patched systemd that nobody knows about). they should be more simple and auditable.
so in short, nothing much we can really do.