Rust devs push back as Serde project ships precompiled binaries
bleepingcomputer.com
bleepingcomputer.com
I found the discussion to be mostly very civilized and focused on finding solutions for those affected.
https://github.com/serde-rs/serde/issues/2538
It's all the right of the author. And he's done so before.
Unfortunately, I'm afraid this could lead to a more fragmented ecosystem which is a pity.
It takes a long time to build trust but it can be destroyed quickly. I'm not sure if the author cares about that though. From my reading of the comments he doesn't.
"If there is implementation work needed in some build tools to accommodate it, someone should feel free to do that work (as I have done for Buck and Bazel, which are tools I use and contribute significantly to) or publish your own fork of the source code under a different name."
https://github.com/serde-rs/serde/issues/2538#issuecomment-1...
This is so sad.
What do you mean by "he's done so before"?
I'm not complaining. It's his project, it's his right. He will be overwhelmed by notifications and this is probably the only way he can stay on top of things.
I don't have to like it either though.
This discussion was cut short by just locking the issues.
The second time it happened to me I stopped our sponsorhip out of frustration. That also sucks and is a bit petty. I acknowledge that.
I really disagree with this characterization of the discussion. While there's plenty of more or less dispassionate comments focused on finding a solution, there's a greater number of people from the peanut gallery drowning that out with comments that amount to "I don't like this!" at best and questioning the integrity of dtonlay at worst. I feel like this demonstrates some of the worst features of open source software on github, where a bunch of nobodies feel the need to add their totally useless input rather than participating in good faith to find solutions and make the software better. You even see comments like that here, speculating that dtonlay is trying to break cargo and force people to use bazel, which is just an insanely conspiratorial and bad faith speculative interpretation that is obviously untrue and helps no one.
Only see comments doing that are on here.
The comments on the github issue are all pretty productive and constructive. The worst is the commentator that calls the approach 'despicable', but also provides constructive feedback and helps describe the problem.
Reddit comments went into tinfoil hat territory.
Discord comments went into calling for dtolnay's head.
Also I'm not lying about (albeit hyperbolic) calls for dtolnay's head. From one of Rust Discords.
don't move goalposts
But ok, let's say limiting yourself only to GitHub comments, you can still find doubt casting and uncharitable interpretations:
> Nice micro-optimization. And if you bury your head deep enough into the sand, doesn't sacrifice anything important either.
> "The only supported way ..." is a polite way of saying "Works for me, WONTFIX".
> Another interpretation of an earlier comment is that he is all-in on bazel/buck and does not care about cargo. Both interpretations have me concerned.
> And ~80 comments with "This doesn't work for me".
Even in this case, the __jem comments make sense. Many people are just writing their version, "I don't like this, " meaning any interesting message is drowned out by noise. THAT will make any maintainer limit to contributors.
I already referenced the worst comment and it falls closer to brutally honest than actually rude.
It's customary to assume other side is arguing in good faith and not making stuff up. I assumed you read all the comments, had access to Github, so you wouldn't have problems finding them. But apparently that wasn't the case.
- https://github.com/serde-rs/serde/issues/2538#issuecomment-1... https://archive.is/bHQS6#issuecomment-1684896969
- https://github.com/serde-rs/serde/issues/2538#issuecomment-1... https://archive.is/bHQS6#issuecomment-1682639314
- https://github.com/serde-rs/serde/issues/2538#issuecomment-1... https://archive.is/bHQS6#issuecomment-1684609445
> I already referenced the worst comment and it falls closer to brutally honest than actually rude.
It's hard to take this at face value when you mistook my quotes for outright fabrication. You either didn't read them all, or you convienently forgot the examples I cited.
Also nice micro-optimization comments is dishonest, apparently dtolnay felt the optimizations provided for his use case were strong enough to warrant a fix.
build.rs absolutely is a glaring security hole in the sense you say, but compared to that, this is much worse. You can verify the build.rs code that you download (at least in theory, and some people in banks or distro packages probably actually do), but binaries are orders of magnitude more difficult to inspect, and with the current Rust build system pretty much irreproducible.
In theory you can compile your own blob, but you'll need musl and whatnot to make a universal Linux build. Code for making the blob is there in the repo.
build.rs is at best equal. It can access your locally available DB, and transmit your data.
We also don't know how it's build. Ideally there is a Docker container out there that does just an import of source code and then builds. No apk install or apt install (you'd do that in a base published layer). Referenced with an SHA256.
We then use this Docker container to pull in the source code AND its dependencies based on a Cargo.lock. Which... isn't there. So we don't know the exact dependencies that went in.
(Even if there were a Cargo.lock, we need to make sure we actually respect it. I believe cargo install by default ignores the lock file and tries to get the latest version that matches).
EDIT: I was wrong, this is not actually `watt` -- it may have been re-using code from the project.
This one of those pile-ons where everyone gets excited about having a cause-du-jour to feel passionate about, while simultaneously ignoring issues that are far more pressing.
The biggest orgs tend to run all of their builds sandboxed, including what happens with build.rs. It's part of how they enforce dependency management day to day, but also helps protect against supply chain attacks.
So not everyone does, but enough people do that you can rely on their complaints for the more well trodden parts of the ecosystem.
Sandboxing doesn't completely prevent supply-chain attacks. You can avoid getting persistent malware on your CI machines, sure. Exfiltration of tokens and secrets during build? Maybe in a small handful of CI setups where admins have carefully split the source fetch, or limited CI network access to Nexus. Exfiltration of tokens and secrets and/or backdooring after the software has been deployed to production? No, build-time sandboxing doesn't help here at all.
On all developer machines as well? No. Very few big orgs do this and only for mission-critical stuff. Some very important ones have docker-based sandboxed workflows or SSH-to-sandboxed-cluster workflows, or air-gapped laptops but that's very, very rare (I worked in air-gapped environment for a bit and it was a massive pain).
Correct, it's more a defense in depth technique, not a complete defense.
> On all developer machines as well? No. Very few big orgs do this and only for mission-critical stuff.
All builds at Google for instance use the model I laid out including 'developer builds'.
That would be extremely useful as the analyzer is a pretty juicy target and also runs proc-macros/build.rs scripts.
[1] https://github.com/bazelbuild/rules_rust/pull/384
[2] https://bazelbuild.github.io/rules_rust/rust_analyzer.html
https://www.grepular.com/Sandbox_Rust_Development_with_Rust_...
Well, that's a bit out of date now as I use podman, to get around the sudo issues. But that's the basic idea.
For lower environments we generally used 3 month or more embargoes on precompiled stuff we couldn’t easily compile ourselves to mitigate some of the supply chain issues if we weren’t directly managing the chain of trust for the binary.
Also it's worth noting that even if it was a sandboxed binary ala https://github.com/dtolnay/watt , it's not obvious that distributions or users would be satisfied with that. For example Zig had this discussion with the own WASM blob compiler that they use as part of bootstrapping. https://news.ycombinator.com/item?id=33915321 . As I suggested there, distributions might be okay with building their own golden blobs that they maintain themselves instead of using upstream's, and that could even work in this Rust case for distributions that only care about a single copy of serde for compiling everything. But it's hard for the average user doing `cargo build` for their own projects with cargo registry in `~/.cargo` to do the same replacement.
I think having an exit path for those that want to compile from source is important, and I can’t understand the reluctance to provide that.
There was also a project (that dtolnay was involved in, I believe!) a few years ago to compile proc macros to wasm.
https://github.com/dtolnay/watt
This would be a lot more compelling if it were integrated into rustc and cargo.
A single person does not audit all of their dependency tree, but many people do read the source code of some if not many of their dependencies, and as a community we can figure out when something is fishy, like in this case.
But when there are binaries involved, nobody can do anything.
This isn't the same as installing a signed binary from a linux package manager that has a checksum and a verified build system. It's a random binary blob someone made in a way that nobody else can check, and it's just "trust me bro there's nothing bad in it".
The developer should be very concerned about what happens if his system(s) are compromised and the attacker slips a backdoor into these binaries-- it will be difficult to impossible to convince people that the developer himself didn't do it intentionally. Their opacity and immediacy make them much more interesting targets for attack than the source itself (and its associated build scripts).
Saving a few seconds on the first compile on the some other developers computer hardly seems worth that risk.
And at the meta level, we should probably worry about the security practices of someone who isn't worrying about that risk-- what else aren't they worrying about?
> Valentini further inquired to the project maintainers, how were these new binaries "actually produced," and if it would be possible for him to recreate the binaries, as opposed to consuming precompiled versions. David Tolnay, who is the primary Serde maintainer, responded with potential workarounds at the time.
The article links to the response [1] as well, which is very good. However, the response doesn't provide workarounds. It provides clear instructions:
>> how is the x86_64-unknown-linux-gnu binary actually produced? Would it be possible for us to re-create the binary ourselves so we can actually ship it?
> By https://github.com/serde-rs/serde/blob/v1.0.177/precompiled/.... Yes.
When the article states that the library author only provided "potential workarounds", it sounded really shady to me, and I believe this is mischaracterizing the real response.
[1] https://github.com/serde-rs/serde/issues/2538#issuecomment-1...
Do they provide binaries for other architectures? Aren't new macbooks and some cheap VPSs using ARM now?
This all sounds like a huge mistake, see python/node deps with C bindings.
"No one" is validating all their transitive dependencies for weird binaries, except a few people, and that's enough to get us all here talking about it. In the internet age it only takes one. But if you make it require more work to inspect a blob, you reduce the chances that you'll get even that one person who sounds the alarm. At minimum malicious code will survive longer that way.
Also, for the “why” - is it just the very slow build times for Rust?
The inability to opt out of running a non-reproducible binary library just to build code. This isn't just devs complaining, it's against several large orgs security policies.
> Also, for the “why” - is it just the very slow build times for Rust?
It's an initial build time thing. It makes no difference except after a full clean or fresh build.
I’m still learning the Rust environment but usually vendoring is what you do when you want full control.
serde-derive is a macro that is used in the compilation of projects. The way the developers have implemented it, when you build a project _from source_, it pulls and executes a binary blob against your code.
The binary in serde-derive is not reproducible (at least, there is no published hash and other users haven't been able to compile the same binary on their computers), and saves a very minor amount of time - serde-derive has to be compiled once per project per compiler version, unless you delete your dependencies.
1. Users may not have the correct build toolchain installed. (Mostly relevant for end-user applications, or native FFI modules.)
2. Building every package from source is slow.
Neither justification makes sense for serde-derive. For 1, serde-derive is just regular Rust code, built by their regular Rust toolchain. You already need to have the toolchain installed to use it! Hell, you need the same toolchain to build the wrapper library for the binary! For 2, serde-derive has a tiny dependency tree, and all of those are likely to be shared by any other proc macros the user depends on anyway, so the only thing you're skipping is... compiling serde-derive itself.
I'd also be curious if anyone has any good explanations as to why a choice like this would be made?
But this isn't a Rust developer issue so much as a distro and security issue. Most Rust developers don't have to concern themselves with supply chain attacks or security audibility, so to them this probably seems like a weird thing to focus on.
But as someone who does have to worry about such things, trading off the ability to audit your supply chain for a small, one-time compilation boost on your first compile is a _terrible_ trade-off.
This is not noisy for no reason - the expectation is all dependencies are compiled from source on your computer, and here the developer forces the use of a binary program, with no alternative.
Yeah. Bazel is the sanitized version of Google's internal build system for most of their projects (known internally as blaze). Buck is Facebook's equivalent.
That said, Bazel and Buck are good for handling projects "at scale" (e.g. company wide monorepos at mid-to-large companies), or with a lot of different needs (e.g. sources in C++, Rust, Python, and Javascript that need to be combined together).
They're not really an alternative to Cargo, and more the next step up for when Cargo isn't sufficient any more. But because they're dealing with much more complicated problems, they're also more complicated to set up and use, so if you're not running into issues with Cargo, you probably don't need to go in that direction (and possibly never will).
Is it worth it? Maybe. Downloading blobs isn't ideal but could be made easier in the future. Ideally a hack like this wouldn't be needed and we'd have comptime serde.
That would be huge if it were a a runtime cost, but this is a one time cost at compile time, and this build will be cached unless you're upgrading serde (or its dependencies). The value of that 10 seconds compared to the cost is a faustian bargain.
So I think the answer is that it’s mostly just noise, but it’s noise you probably don’t want to have to deal with unless you’re invested in Rust itself.
It has to do with how Rust builds are structured.
The compilation unit is the crate, and the result of compiling it depends on the source code, the version of hte compiler, the compiler options, the target triple, and the set of features (think #ifdef FEATURE_FLAG).
Since there's exponential blowup of possible combinations of a single crate, the largest package registry (crates.io) does not cache compiled crates. It only caches their metadata, and when Cargo (the package manager) does dependency resolution, it uses that metadata alone - and your build system (also Cargo, most of the time) will need to actually build all those artifacts.
Seems pretty reasonable, right?
There's a special kind of crate called a proc_macro crate (procedural macros). When you use these crates you're not adding their code to your compiled output - you're creating a compiler extension - the compiled crate is loaded by the compiler, and then used to generate code in your crate, which is then compiled by the compiler.
One of the reasons Rust build times (particularly clean builds) is that they first have to compile all these compiler extensions and then use them to generate the code that the compiler compiles. These crates are super useful however, like serde, which generates serializers/deserializers that can be used for a wide variety of encodings (XML, JSON, TOML, various binary encodings, and so on).
It would seem that the same rules apply for normal crates - there's exponential blowup of compiler version, target triple, and features - but that's not really true. proc_macro crates usually don't have features, the compiler ABI is stable enough that they probably won't break across versions, and almost everyone and their mother is compiling on a handful of host systems (x86_64-linux being the most important one).
So while it's true that crates-io probably shouldn't cache every crate for every permutation of compiler/target/feature/etc, a few crates are so pervasive and used in such a sparing manner that it should be possible for the registry to host those
In fact some build systems can do this automatically. But Cargo can't. But it does have a pre-build feature (build.rs) that works like proc macros, and can pull down a pre-compiled proc macro crate, so widely used ones can use this hack to get around limitations of cargo.
My personal take is that the reason that crates-io and Cargo haven't been upgraded to do this, despite years of people complaining, and known solutions existing, is mostly personal/political. There aren't many people willing to put in the work to do it, and the teams haven't been open to contribution of this kind for a long time. For example, they do not believe that Cargo is a build system, and a lot of this is "out of scope" for cargo.
serde = { version = "1.0", features = ["serde_derive", "fast_compile"] }
Either opt-in or opt-out is fine. It's surely useful during development but it shouldn't be on for production builds. Any transitivity issues should be considered problems with the dependency.
There's also the problem that the build may fail because it will still download the binary, which you may want to forbid entirely.
https://doc.rust-lang.org/cargo/reference/resolver.html#feat...
That said I'm not sure how that applies transitively, I could have sworn I've seen conflicting features not cause transitive build failures.
Also, the “security” guy claiming no one is going to read it, do you read linux kernel every update? No, not you, but other developers do, or whenever someone is interested/auditing they can, that kind of arguments are stupid.
I don’t see a problem with the community forking as motivation to get the maintainer to come back the conversation or to provide a path for the community to resolve the issue for themselves. A fork was literally one of dtolnay’s own suggestions, bluff or not.
Me too. As a Rust user and small-scale evangelist, I find this whole thing a bit interesting and also somewhat sad. At least, it seems like increased awareness, and perhaps sandboxing, of proc macros are good outcomes that may result.
> how can you work something out when the maintainer can and has already just locked the issue and walked away?
That's not what happened here. dtolnay has not walked away, and the discussion is progressing. For instance, this pre-RFC [0].
> I don’t see a problem with the community forking
Serde, to the limited extent that I understand it, works because it is a single shared interface - multiple forks would be counter to the purpose of the project. That said, here we're talking about serde-derive specifically, so it's not quite a fork of the serde interface that we're talking about. But, I can understand why people might be upset that such an important a piece of the Rust ecosystem is such a flashpoint.
[0] https://internals.rust-lang.org/t/pre-rfc-sandboxed-determin...
But, I think the attention on proc macro performance and overall weaknesses in security with cargo could be net benefits, even if I don't personally care about a 9 second speedup in clean builds. I was also really dreading dealing with serde forks, selfishly.
146 kB: https://crates.io/crates/serde_json
43 kB: https://crates.io/crates/sj
do you know that, or are you just assuming that?
In my opinion it's better to have simple interfaces for complex features, rather than simple interfaces for simple features. You could code those yourself.
Most crates do not raise their major version when increasing the minimum supported rust version. So there is precedence for treating which changes can break for some people, but which don't have a breaking API changes, as non-breaking for semver purposes.
I don't mind it for when I work in Node - honestly I prefer it. But in Rust? The entire toolchain is designed to compile from source. It may be a bit slow for certain crates still, but that's much weaker justification to make pre-compiled the default.
Crates should compile from source by default as long as source actually exists (ex. has not been lost somehow, or wide distribution of the crate is unlikely). I don't buy any of the malicious theories since the author is well appreciated and respected. It's likely just a misguided choice in this case.
People have rightly mentioned build.rs flexibility - I think there is nuance here and maybe it's good to discuss what default behavior should be expected by users, since sometimes build.rs does wild and useful things.