Is the solution against such attacks in the future only to scrutinize more, or are there other reasonable options in terms of hardening?
On the flip side the elegant/readable build system means that the place this exploit was hidden wouldn't exist. Though I wouldn't confidently say that 'no hiding places exist' (especially with the parts of the ecosystem that wrap dependencies in other languages).
How would an automated CI or build infrastructure stop this attack? It was stopped because the competent package maintainer noticed a performance regression.
In this case, this imagined build system would have to track every rust library used in every package to know which packages to perform an emergency release for.
Rust isn't really the point here, it's the age old static vs dynamic linking argument. Rust (or rather, Cargo) already tracks which version of a dependency a library depends on (or a pattern to resolve one), but it's besides the point.
Not to mention if you have a Rust application that depends on C libraries, it already dynamically links on most platforms. You only need to rebuild if a Rust crate needs to be updated.
Cargo already has this information for every project it builds. That other systems do not is their issue, but it’s not a theoretical design.
A few things:
1. It'd be Cargo.lock
2. Debian, in particular, processes Cargo's output here and makes individual debs. So they've taken advantage of this to already know via their regular package manager tooling.
3. You wouldn't dive into and look through these by hand, you'd have it as a first-class concept. "Which packages use this package" should be table stakes for a package manager.
> Then what? Do I fork each one, bump the version, hope it doesn't break everything, and then push an emergency release!??!
The exact same thing you do in this current situation? It depends on what the issue is. Cargo isn't magic.
The point is just that "which libraries does the binary depend on" isn't a problem with actual tooling.
People already run tools like cargo-vet in CI to catch versions of packages that may have issues they care about.
False. In the current situation, you just release a new shared library that is used system-wide.
Oh I 100% agree with this, but that's not what was being talked about. That being said, I don't think the distribution model is perfect either: it just has a different set of tradeoffs. Not all software has the same risk profile, not all software is a security boundary between a system and the internet. I 100% agree that the sheer number of crates that the average Rust program pulls in is... not good, but it's also not the only language/platform that does this (npm, pypi, pick-your-favorite-text-editor, etc.), so soloing out Rust in that context doesn't make sense either, it only makes sense when comparing it to the C/C++ "ecosystem".
I'm also somewhat surprised that the conclusion people come to here is that dynamic linking is a solution to the problem at hand or even a strong source of mitigation: it's really, really not. The ability to, at almost any time, swap out what version of a dependency something is running is what allowed this exploit to happen in the first place. The fact that there was dynamic linking at all dramatically increased the blast radius of what was effected by this, not decreased it. It only provides a benefit once discovered, and that benefit is mostly in terms of less packages need to be rebuilt and updated by distro maintainers and users. Ultimately, supply-chain security is an incredibly tough problem that is far more nuanced than valueless "dynamic linking is better than static linking" statements can even come close to communicating.
> A problem like this would propagate much faster. Here the threat actor had to work hard to get his library updated in distributions and at each step there was a chance that this is detected.
It wouldn't though, because programs would have had to have been rebuilt with the backdoored versions. The book keeping would be harder, but the blast radius would have probably been smaller with static linking except in the case where the package is meticulously maintained by someone who bumps their dependencies constantly or if the exploit goes unnoticed for a long period of time. That's trouble no matter what.
> Now think about a Rust package automatically pulling in transitively 100s of crates.
Yup, but it only happens at build time. The blast radius has different time-domain properties than with shared libraries. See above. 100s of crates is ridiculous, and IMO the community could (and should) do a lot more to establish which crates are maintained appropriately and are actually being monitored.
> Sure, a distribution can later figure out what was affected and push upgrades to all the packages.
This is trivial to do with build system automation and a small modicum of effort. It's also what already happens, no?
> But fundamentally, we should minimize dependencies and we should have quality control at each level
Agreed, the Rust ecosystem has it's own tooling for quality control. Just because it's not maintained by the distro maintainers doesn't mean it's not there. There is a lot of room for improvement though.
> (and ideally we should not run code at build time). Cargo goes into the full opposite direction. Rust got this wrong.
Hard, hard, hard disagree. Nearly every language requires executing arbitrary code at compile time, yes, even a good chunk of C/C++. A strong and consistent build system is a positive in this regard: it would be much harder to obfuscate an attack like this in a Rust build.rs because there's not multiple stages of abstraction with an arbitrary number of ways to do it. As it stands, part of the reason the xz exploit was even possible was because of the disaster that is autotools. I would argue the Rust build story is significantly better than the average C/C++ build story. Look at all the comments here describing the "autotools gunk" that is used to obfuscate what is actually going on. Sure, you could do something similar for Rust, but it would look weird, not "huh, I don't understand this, but that's autotools for ya, eh?"
To be clear, I agree with you that the state of Rust and it's packaging is not ideal, but I don't think it necessarily made wrong decisions, it's just immature as a platform, which is something that can and will be addressed.
Only a few projects are built as system-wide libraries that expose a C-compatible abi interface; rsvg comes to mind.
https://github.com/tukaani-project/xz/commit/6e636819e8f0703...
Arbitrary code downloaded from the internet and run at build time? That's a nightmare scenario for auditing, much worse than anything Autotools or CMake can offer.
> not requiring me to be an expert in another language just to audit it.
Do you do that every time your Cargo.lock changes?
Says who? Make is just as good at calling arbitrary code as Cargo. Including code that reaches out over the network. Have you audited every single makefile to ensure that isn't the case?
Whereas building my crate can run code locally that no one has ever audited.
Again, we're not talking about the dependencies that I choose, but the whole transitive closure of dependencies, including the most low-level. Did you examine serde the first time you used a dependency that used it? serde did have in the past a slightly sketchy case of using a pre-built binary. Or the whole dependency tree of Bevy?
I mean, Rust has many advantages but the cargo supply chain story is an absolute disaster---not that it's alone, pypi or nodejs or Ruby gems are the same.
Fedora packages a large number of Rust libraries, just as you describe. Nothing prevents you from using the packaged libraries if you prefer them.
You may find helpful information here: https://docs.fedoraproject.org/en-US/packaging-guidelines/Ru...
Nothing except, in no particular order: 1) only having one version of crates 2) mismatched features 3) new transitive dependencies that can be introduced at any time without any warning 4) only supporting one version of rust 5) packages being noarch and basically glorified distro-wide vendoring—so their build.rs code is still run on your machine at cargo build time
Same as any other library provided by the distribution in any other language.
> 2) mismatched features
Same as any other library provided by the distribution in any other language.
> 3) new transitive dependencies that can be introduced at any time without any warning
Not in packaged Rust libraries in Fedora, at least. Please read the aforementioned link.
> 4) only supporting one version of rust
Same as any other library provided by the distribution in any other language.
> 5) packages being noarch and basically glorified distro-wide vendoring
Packages containing only source is a consequence of the Rust ABI still stabilizing, see: https://github.com/rust-lang/rust/pull/105586 After ABI stabilization, Rust libraries will be first class like any other language.
Building packages with up-to-date dependencies is also vastly preferable to building against ancient copies of libraries vendored into a codebase at some point in the past, a situation I see far too often in C/C++ codebases.
The fundamental problem here was a violation of chain of trust. Open source is only about the source being open. But if users are just downloading blobs with prebuilt binaries or even _pre-generated scripts_ that aren't in the original source, there is nothing a less-obscure build system will save you from as you are putting your entire security on the chain of trust being maintained.