NixOS and reproducible builds could have detected the xz backdoor
luj.fr
luj.fr
> I am a NixOS developer and I was surprised when the backdoor was revealed to see that the malicious version of xz had ended up being distributed to our users.
As always theory and reality are different, and the thing that made xz possible was never a technical vulnerability with a technical solution—xz was possible because of a meatspace exploit. We as a community are very very bad at recognizing that you can't always just patch meatspace with better software.
Nix declarativeness is quite useful to increase protection against exploits in a number of ways. Unfortunately, there is still a lot of untapped potential. My number one priority would be to implement fine-grained ephemeral containers. Guix has these already.
This would make it convenient to run every single process with restricted privileges, including no access to ~/, except those directories that are needed by the task. That would prevent e.g. a rogue pip package from stealing SSH keys.
Still, I think the xz backdoor did not work on NixOS because its unusual non FHS-compliant filesystem structure.
Right, but this is not part of the security model, it's an incidental attribute of the OS that's there for other reasons and easily solved for if the attacker had prioritized it. The only reason why it didn't work is because the attacker didn't bother making it work on NixOS, not because he couldn't have if he'd wanted to.
It didn't work on NixOS because the build-time hooks that inserted the backdoor only activated itself when it recognized that it was being built for an RPM or Debian package.
Unfortunately, this wouldn’t help with the xz vulnerability because the SSH server is the one loading the compromised library in that case (indirectly). Since SSH itself needs to have access to the private keys, it’s not really easy to secure it against vulnerabilities in the library it loads itself.
On the flip side, unless the vulnerability is in one of the important binaries/shared libraries, the amount of damage it can cause it probably quite contained with simply having good user isolation. Nix can make this analysis really simple (because of explicitly specified dependencies), so you can crack down on critical dependencies a lot more easily.
Sure, but I think that leaves out many use cases. What if I want to e.g. start a Python shell that has access to certain directories, and nothing else, including no network access?
Nix provides a good way of doing that for common use cases, as it has decent support for Firejail. But I would like something like Guix containers, which is convenient for any ad hoc use case. This greatly reduces any security threat. It's a poor-man's QubesOS.
Does Firejail handle dynamic access (eg. I may want an xz invocation to work on my private keys, but not THIS specific one where I’ve given it a completely different file?).
I quite like pledge/unveil for this kind of thing on OpenBSD, although that’s for a different threat model.
But Nix relies on ephemeral shells and flakes, and they don't play so well with each other. The interface is clumsy. Guix, in contrast, has a pretty nice set of CLI switches for these features.
Even normal distros should prioritize some simple graphical UI for this. Running programs with minimal privileges would result in a significant enhancement of security. The kernel features for achieving this are already there.
bwrap does not require SUID, it only needs it if user namespaces are disabled for unpriviledged users.
Like if I were to try and find a not annoying way to do this, it would be to snapshot and overlay mount my filesystem at process launch time, then give a heuristic warning at process termination about what was changed.
But then we're still into basically SELinux territory which because of course there's upfront things we don't want to allow read access to - i.e. SSH keys.
(don't know what the lesson here is other then "for the love of god could we get a standardized secrets filesystem and/or API or something". Looking at you Hashicorp Vault and ~/.vault-token).
In a regular distro, it is tolerable, but a better UI would be great.
This is such a headache with snap and flatpak though.
If you're trying to do something that the package maintainer thought of in 5 minutes of testing then it's usually fine, but still inside any non-trivial application you'll often find parts that try to use extra privileges that aren't documented because they're not normally considered "privileges".
Some examples of sandboxing issues:
* FreeCAD doesn't have access to /usr which means when you try to make a Draft ShapeString you can't pick any fonts
* FreeCAD stores the path to the shape of a milling cutter inside your project, but that path is inside /mnt/.FreeCAhjhffg or whatever so it doesn't work after you restart the program
* Inkscape gcodetools has its own "idiomatic" way of saving out results which doesn't use a file dialog, and therefore can't save any gcode because it can't write to your home directory
* Matrix is only allowed to access ~/Downloads/ which means you can't share any files with anyone unless you copy them into Downloads first
* "Recent" in the Gimp file picker doesn't show any of my recent files, presumably because it is using its own sandboxed file dialog which doesn't have access to the actual "Recent" files
* Docker can't access /tmp/.X11-unix which means you can't give docker containers access to X
In all of these cases you can work around it of course (mainly by having an accurate mental model of the problem and guessing a path that the program is allowed to access), but the user experience is just made worse for no benefit.
The general theme is that the user wants the sandboxed program to be able to do something that the person who assigned privileges didn't think of.
So maybe if we must do sandboxing, let's make it easy for users to break programs out of the sandbox when it suits them?
As a user, I quite like the iOS approach, of apps "sharing" their ressources with other apps or asking for permission to access this or that collection of ressources. This can probably be improved and adapted to a non-touch model, but I think the concept is nice.
But, of course, apps have to be built for this kind of environment, so I think there will unfortunately be some janky transition period, with the customary competing, incompatible implementations.
It actually ships with rulesets for hundreds of programs that tend to be quite polished and work out of the box.
Personally, I dislike flatpak because it doesn't let me control the dependencies of packaged software, and I feel we loose one of the most important advantages of Linux.
> let's make it easy for users to break programs out of the sandbox when it suits them?
At least for Flatpak given the things you described this is quite straightforward via bind mounts. Although it did seem a bit goofy having an entire list of per-application bind mounts in my fstab. Maybe things have improved since I last tried going that route?
> "idiomatic" ... doesn't use a file dialog,
That's an Inkscape (plugin?) bug plain and simple. GUI apps should be using the appropriate xdg portals for all supported operations at this point. The only excuse (IMHO) is for missing or broken functionality.
It would be like a system tray widget not working and blaming the DE instead of the program that fails to implement the decently old and widely adopted standard.
> Matrix ... you can't share any files with anyone
Which client is this? Anyway the xdg portal should work. Did the dev try it? Programs should never need blanket access to save or open specific user supplied paths as a one off. That's a large part of the point of sandboxing stuff in the first place.
We can state what they "should" be doing until we're blue in the face, but if they "aren't" doing that then it doesn't help.
> Which client is this?
The Matrix one is from a long time ago. It could have been Element? Or Riot? Were they the same thing? Don't know. I only used it briefly.
True enough. To be clear I don't fault anyone for not using new or known to be broken or little known standards.
However at some point responsibility has to shift to the developers. There are standard ways of doing things. Just as you can't expect an arbitrary project to support your pet API that hardly anyone uses, developers can't expect major distributions or the majority of users to cater to their refusal to conform to widely accepted standards.
I'm not saying you shouldn't use a particular program. Just that I don't think it's reasonable to fault the tooling for certain things.
Please no. I understand why Flatpaks do it, but this is one of the most ridiculously annoying things about the Flatpak sandbox. You can often only drag 'n drop from ~/Downloads/, and from any other location either causes the receiving application to glitch out, fail silently, or fail with a general error. Hell, you sometimes can't even copy-paste an image from one application to another via the copy-paste buffer! Meanwhile on macOS and Windows it works perfectly.
I think launching e.g. a Python shell with some packages that are potentially compromised and letting those read ~/.ssh and whatever else they want is fundamentally insecure. Rogue PyPI packages that steal SSH keys is not a theoretical security breach, it already happened several times [1].
The current security model in Unix is untenable. But I agree well-implemented sandboxing should be frictionless. What you are experiencing is probably a X or Wayland sandboxing glitch. I also dislike Flatpak, for other reasons, but that doesn't make sandboxing a bad abstraction. It's just that we don't happen to like this particular implementation.
[1] https://www.packtpub.com/en-tw/learning/tech-news/python-lib...
The best solution would be a framework akin to macOS that pops up 'allow application X to access folder Y from now?' in the UI or in the CLI as a terminal prompt whenever an application tries to access a folder. With a special permission for "full home access" and "full disk access".
Drag and drop could be made to always work since it's being done by a user. Request this feature from your operating system's developer.
That should be a privileged status. If you manage to trick the user into installing malicious software followed by granting it elevated privileges then you likely didn't need such a roundabout method in the first place.
Hopefully one day people will finally accept that this is bad design and that humans require compromises on security
Breaking or making actively annoying expected and useful functionality isn't security (and has a long track record of leading to workarounds which compromise security).
See also https://xkcd.com/2044
Maybe Linux isn't as effected now because it is not very popular as a desktop system. But this issue will have to be addressed as/when Linux becomes more popular. Having networked clients that do image parsing, etc. (usually in C code) without any sandboxing will just lead to mass exploitation, data exfiltration, etc.
The Linux desktop has to move away from the 90ies security model where the internet was relatively safe and attackers would only be after UID 0.
a lot like MacOS lately asking me "(AppName) wants to access your Downloads folder, cancel or allow?" when I have just directed it to open a file.
I don't think it asks that when it goes through a portal (e.g. file dialog)?
It didn't work on nixos because the build-time check included checking whether the build was being executed in a debian or fedora build environment. This was to avoid suspicious build failures on distros with weird toolchains or incompatible architectures/ABIs/library versions. (The backdoor was a precompiled .o file so rather ABI sensitive)
You would essentially need Android or iOS for it to not be a pain in the ass
Nonetheless, in this year and age this should be the bare minimum from a security point of view.
I don't have hard data, but my impression is that the general tendency in Flatpaks is that they are able to do more sandboxing over time. When Flatpak was new, a lot of applications pretty much required completely opening their sandboxes, the same applications have much more limited privileges nowadays.
It's a long process, but at least with desktop applications there is progress. Unfortunately, the same can't really be said about command-line tools and development tools (NPM, cargo, pip, editor plugins, etc.).
For CLI a solution would probably look a lot like unshare or (a more user friendly version of) setcap. The user would need to reach out to the sandbox to communicate what additional things to permit during this specific session.
And then inevitably someone would configure the equivalent of passwordless sudo at which point I wonder what the point of the whole thing was to begin with. Related, we need a better paradigm for CLI to differentiate between user versus programmatically generated input. A program shouldn't be able to impersonate me unless I explicitly granted it some extremely unusual privileges.
This. Which is why I'm using Qubes OS designed around sandboxing, and I can't recommend it enough.
Besides how large the package repos are, what other reasons would one chose Guix over NixOS?
Otherwise, available packages are tidy and well defined.
NixPkgs, which I use and develop for, is less tidy.
Also, I'm not a NixOS critic either: I'm writing this from NixOS! I just don't think there's such a thing as a security cure-all as long as humans are in the loop anywhere.
Thank you very much for citing that! along with highlighting the fact that the exploit was in fact, not detected by reproducible builds prior to other means of discovery.
In recent times, actual reality is often maligned when compared to how people feel about objective reality and how it meshes with their individual value systems.
I have personal values too, but I don't hold the opinion that actual reality is less significant than how I feel about it. It's not a popular perspective 8-/
I've always liked to say: The difference between theory and reality is that, in theory they're the same, and in reality they're not.
I hope the realization that the reproducible builds of NixOS _could_ have detected the xz exploit, but didn't, will lead to new advances in the analysis of those reproducible builds to detect other exploits sooner in the future.
Also I (as a nix user myself) think it's unlikely NixOS would have caught it. As evidenced by the fact that it didn't. (Yeah I realize I just said next time it might happen differently but it'd be foolish to put faith in nix without evidence).
they are relevant.
and if any OS, then for windoze please!
There.
Of course this doesn't buy you anything if the exploit is changed to target a "patched" OS but the same goes for the proposed Nix solution.
automatic here means that code written prior to the existence/knowledge of the xz backdoor catches not only this specific attack, but the entire class of such tarball attacks.
I don’t see what this solves though. Couldn’t a malicious maintainer simply add binary blobs directly to the source code repository?
The author suggests Github is trusted, as though Github validates code in some way. Which of course it does not.
Verified reproducible builds could have countered the xz utils break, SolarWinds Orion subversion, and many others. It's worth doing.
A binary blob by itself is harmless, you need something to copy it over from the build env to the final binary. So it's "safe" to ignore binary blobs that you're sure that the build system (which should all be human-written human-readable, and a small portion of the total code in sane projects) never touches.
That said, of course, there's still many options for problematicness - some projects have commit autogenerated code; bootstrapping can bring in a much larger surface area of things that might copy the binary blob; and more.
There's also value in leaving a trail to make auditing easier in the event that an attack is noticed or even if there is merely suspicion that something might be wrong. More visibility into internal processes and easier UX to sort through the details can easily make the difference between discovery versus overlooking an exploit.
If you’re downloading and executing a binary from github releases, then you’re completely at the mercy of the maintainer (nix only does that with closed source packages)
The author demonstrates that Nix can be configured to generate the tarballs from git that go into building the binaries.
What I don't see, however, is how is this a feature that requires Nix or NixOS?
Any build system out there (including the stuff that goes into RPMs and Debs) can be configured to generate tarballs as a intermediate step.
In fact making reproducible builds is a major thing that Debian has been working on for some time now.
* the NixOS build process was unable to perform a full-source build of xz because xz is required too early in the bootstrap;
* a proposed adjustment to nixpkgs to automatically detect compromises of nixpkgs dependencies which are required early in the bootstrap.
Other ecosystems can of course also attempt full-source builds and discover the discrepancy; the entire point of the article is that nixpkgs currently cannot.
Ubuntu on ZFS can do this as well.
As for ZFS... Dealing in filesystem snapshots is comparatively a bit awkward. If you want to recreate that config elsewhere you have to move the whole snapshot rather than just the recipe for building it, and even then it'll break if the system architecture is different on the target machine. If you've got two of them (perhaps labeled "good" and "bad"), you're not going to get anything friendly when you try to diff them, nor is there an obvious way to use things like `git bisect` to reason about where the problem occurred.
None of these things are show stoppers, but working with code that defines some state is just so much easier than anything you get out of a filesystem which happens to remember that state, but can't tell you why it should be the way it is.
That wasn't the highlight of the point. It was that you can restore to a known good version of the operating system, effortlessly, regardless of what the operating system is. It could be Ubuntu, Nix, or FreeBSD. Broken OS, select an older snapshot in the boot loader, and you're golden again.
As much as I want to put all the power in the hands of the user, I'm sympathetic to the plight of the IT guy who has a team that always boots to "last known good" config thats several years old because they just don't trust updates in general.
Do people even read package derivations? Feels like it'd be easy to check in a derivation with an exploit.
Building from the git commit the release claimed to be from would result in a different binary than building from the tarball if the environment check passed.
The problem is just that the build systems did not strip pre-compiled object files before building from source. Even with that fixed, if nobody checks the source code then you can add all the backdoors you want, and there is nothing in NixOS or any other distro that would protect against that.
It points out the need and use for build-manager tools that go a step beyond union file system layers, but track then enforce that e.g. tests cannot pollute build artifacts. Take a causal trace graph of files affecting files, in the build process, make that trace graph explicit, and then build a way to enforce that graph, or report on deviations from previous trace graphs.
the article demonstrates how to imorove NixOS (or Guix) to automatically catch any such discrepancy in the future.
> the maintainer provided tarball was honestly generated from the original source code.
How, then? What about differing versions, etc. or has it been mentioned and I just missed it?
Just make sure the generated tarball can be generated from the source code itself, do not exclude anything, git add & commit everything. Can't we do that? We would still have to look at commit history in this case, I believe, and again, he said it himself, it was harmless to the naked eye, so even then, how could we verify? Maybe I don't understand what he meant by verification, but if maintained tarballs are generated from the owner's source code and is not on GitHub (or anywhere else, just a git repo), that is a problem in itself.
Of course there was more to it than just pushing poisoned test files, but still. I do not see how Nix would have prevented it, if the git repo has those test files and with seemingly harmless code (and is reproducible).
Perhaps what we can do is: if an (in)famous project has changed its main lead, then pay closer attention to the commits and check who it is? I don't know, TBH.
Did I misunderstand the article, or am I missing something?
> To build xz from sources, we need autoconf to generate the configure script. But autoconf has a dependency on xz!
Both directions of this seem crazy to me.
1. Why the heck should a build configuration tool like autoconf be unable to function without a compression tool like xz? That makes no sense on its face.
2. For that matter, why the heck should xz, a tool that is supposedly so fundamental, have a hard dependency on a boilerplate generator like autoconf?
At the end of the day all autoconf is doing is telling you how to invoke your compiler. You ought to have a way to do that without the tool, even if it produces a suboptimal binary. If you care about security, instead of taking a giant tarball you don't understand and the running another tool in it, shouldn't you just generate that command line somehow (even in an untrusted fashion), review it, and then use that human-verified script to bootstrap?
And if you need a (de)compressor that low on the dependency tree so that literally the entire world might one day rest on it, surely you can isolate the actual computation for bootstrapping purposes and just expose it with just the open/read/write/close syscalls as dependencies? Why do you need all the bells and whistles?
At face value, both autoconf and its cousin pkg-config are overly complex dogshit software - both with circular dependencies - that should have been retired long ago in favor of something else. I scream with joy when I use software that uses its own bootstrapper or cmake.
Before you think "but I've never had this problem, you must be bonkers" - try building software on a fresh Solaris box with no GNU anything installed and you need to install one of these monstrosities with their circular dependencies. Your hair will fall out before you're done.
> At face value, both autoconf and its cousin pkg-config are overly complex dogshit software - both with circular dependencies - that should have been retired long ago in favor of something else. I scream with joy when I use software that uses its own bootstrapper or cmake. Before you think "but I've never had this problem, you must be bonkers" - try building software on a fresh Solaris box with no GNU anything installed and you need to install one of these monstrosities with their circular dependencies. Your hair will fall out before you're done.
I've used & seen plenty of the mess of autoconf, thank you. It's a hell I don't want to go back to, and it's a hell a lot of people successfully avoid. But even then, I've also never noticed it requiring compression or decompression, which is partly what boggled my mind at the statement.
In any case, the question was: why should autoconf have a hard dependency on xz? Your response to that was autoconf is complicated and has circular dependencies? How is that a response? That was the premise of the question, not the answer.
For the case of xz not using the upstream-generated configure would probably be doable with some effort but doing the same for glibc, gcc, gnumake etc. would be much more difficult.
(Autoconf is a pain and I would try to avoid for new projects, but for detecting all kinds of crazy old unices I'm not sure what is better)
xz targeted deb and rpm. The vast majority of what is facing the world.
Nix did not stop it.
I believe this article feeds the possible vuln rather than prevent it.
This is true for every distro and it grinds me that Nix is even mentioned.
A lot of issues do get undetected. E.g. the Debian OpenSSL security accident was only detected when a gazillion servers had predictable SSH keys.
I've not seen this many from multiple but evidently related green accounts before. Given the implications about nation state actors in play, it's tempting to jump to conclusions here.
You can add these kind of lines:
news.ycombinator.com##tr.athing.comtr:has(a.hnuser:has-text('banana_dick'))
news.ycombinator.com##tr.athing.comtr:has(a.hnuser:has-text('sirspamalot'))
> It's quite amazing that these new "green" accounts can post as many messages as they want while I'm being restricted
That's not what is happening .. look at the account names and the number of comments made by each.
Thanks, I didn't know this was something we could turn off. Although it does feel like they could just default to collapsed too.
Comparing the tarball's contents against the VCS repository would've likely made this easier to catch, but at that point you might as well just use the VCS repository directly.
a public, deterministic and assured build facility for oss would be cool. also maybe a deprecation of autotools.