Serde has started shipping precompiled binaries with no way to opt out | Hacker News Reader

Serde has started shipping precompiled binaries with no way to opt out

github.com

98 pointsdavikr25 comments

And no justification or explanation that I can see. David Tolnay's response:

> Thanks for the comments everyone. I'll go ahead and close this. The precompiled implementation is the only supported way to use the macros that are published in serde_derive. If there is implementation work needed in some build tools to accommodate it, someone should feel free to do that work (as I have done for Buck and Bazel, which are tools I use and contribute significantly to) or publish your own fork of the source code under a different name. Separately, regarding the commentary above about security, the best path forward would be for one of the people who cares about this to invest in a Cargo or crates.io RFC around first-class precompiled macros so that there is an approach that would suit your preferences; serde_derive would adopt that when available.

I don't know what any of that means. I need an ELI 5

Serde is a Rust library for serialization. serde_derive allows creating serialzers/deserializers automatically by marking structs. It uses Rust's procedural macros to generate code at compile time. Rust compiles procedural macros at compile time before compiling rest of code. To speed up compillation of code using serde_derive, a recent change was made to implement the procedural macros with a precompiled binary

So is this just because server is known to make rust really slow to build?

I'm curious why rust doesn't just compile your binaries once and then you wouldn't have to compile them on each build

It generally does cache built crates for future builds on the same system, so it is usually only the initial build on a given system and the build after dependencies are updated that requires rebuilding serde_derive. I honestly don't know why they thought it was worth doing.

It very much does this. The compile times are hardly a concern to the vast majority of rust devs in part because of that cache.

serde is also used in a LOT of projects and pulled in as a dependency or subdependency often in the crates ecosystem.

For organizations with internal transparent reproducible build guidelines or similar rules, this maybe complicates things and introduces uncertainty.

So much uncertainty nowadays. It's exhausting.

You can take the generous / positive view that this will encourage formalizing reproducible build policies in organizations.

Water is wet, the sky is blue.

I'll be pinning my versions of serde until this gets reverted, fixed, or forked. It's an extremely surprising move to disregard supply chain concerns in favor of non publicly benchmarked performance improvements.

Correction: It's benchmarked, https://github.com/serde-rs/serde/pull/2514.

This is a pretty solid way to motivate the Rust core team to finally solve the proc macro issue. We've been living with the slowness for so long. Give us wasm and precompiled, sandboxed binaries from crates.io!

I suspect dtolnay is trading a bit of his outsized influence on the community [1] to light a fire, and I think it's working.

[1] https://crates.io/users/dtolnay?sort=downloads

I think the much bigger issue this spotlights is that many foundation crates are maintained by random third parties and not the rust organization. I think this will turn into Rust's "leftpad" incident, unless dtolnay rolls back this change soon.

dtolnay is not random by far -- he's one of the most influential crate authors in the ecosystem. If anyone is trustworthy enough to pull a stunt like this, it's him.

I would challenge you to find more than a half dozen crate authors with more downloads:

https://crates.io/users/dtolnay?sort=downloads

These are not `leftpad`-level crates. This is fundamental tech in the Rust ecosystem, especially in proc macro work.

> If anyone is trustworthy enough to pull a stunt like this

I'd turn this around. If somebody as trusted as him, pulls such a stunt, how are we supposed to trust anybody?

> These are not `leftpad`-level crates.

They are like leftpad, because they're very popular (indirect) dependencies outside the control of the rust organization.

If you find malware in any package he publishes I'll bring out the pitchforks but this drama is all pretty much all hysteria at this point.

Calling his contributions leftpad is pretty rude, considering the massive difference between complexity and importance of the two

I've only had a handful of interactions with him and he's been a stand-up guy. I'll value the human over the outrage here.

You're completely misunderstanding what the guy you're responding to is saying. He is not comparing the contributions of the guy to the developers of leftpad. He's saying that the Rust crates system is vulnerable to issues.

Left-pad was un-published from NPM and broke builds using it. Serde-derive includes unreproducible and unoptional binaries, breaking many builds that prohibit such well trodden attack vectors. What's the difference?

Trust is easy to destroy and hard to build. This is trust destroying behavior. There may be some positives that result from this but his reputation may not recover and his motivation for this behavior may not get implemented, it's more likely that cargo just gets updated to block this behavior (good for all of us, but not the intended outcome).

> Give us wasm and precompiled, sandboxed binaries from crates.io!

Even a precompiled, sandboxed WASM binary is already too opaque. Sure, it cannot directly do bad things to the build machine (because it's sandboxed), but it still could inject malicious code into the final output of the compiler. And since it's a binary, it cannot be easily audited (through things like cargo-vet), unless it can be re-created from its source code through a reproducible build step.

And if it can be re-created through a reproducible build step, then it's not much more than a cache. Which points to the true issue being deficient caching of some intermediate build steps; distributing precompiled binaries should not be actually needed.

You're acting like the infrastructure for reproducible builds can't be built in Rust and then added in to tools like `cargo vet`. There are lots of options here, including having the proc macro wasm files built on crates.io.

There is a reason why Rust doesn't yet have these sandboxed, pre-built macros and why the community needed a kick to get working on it. It's not easy, but it's entirely possible to reach the bar of auditability.

You realize that the first step people do when developing Rust is use a pre-compiled rustup to get a pre-compiled Rust binary toolchain, right?

> You realize that the first step people do when developing Rust is use a pre-compiled rustup to get a pre-compiled Rust binary toolchain, right?

Not necessarily, they could use their distribution's package manager to obtain the Rust compiler (which is what I do). Or they could use an older Rust compiler to build rustc and cargo from source (which is what I did back when Rust was newer). Yes, you cannot avoid having a pre-built compiler somewhere in the chain, but we should be working to minimize the number of these hard-to-audit dependencies, not increase them.

Judging by the response I've seen the only thing he's lighting on fire is all the trust he's built up.

If compilation is slow, use a binary cache or packages from your distro.

Mixing binaries with source code means the whole package is not FOSS anymore.

It is illogical to ship binaries for only part of a crate. Shipping binaries for the whole crates is much simpler and gives a much larger speed-up.

But ship the binaries separately from the source code.

> Mixing binaries with source code means the whole package is not FOSS anymore.

This is completely untrue and a bad-faith argument. The source is right there for the binary. It's still FOSS.

The whole package is not FOSS. Individual files might be.

You are right that my assumption in this unusual arrangement is that the binary does not necessarily correspond to (the same version of) the source code.

serde is bloated anyway. see for yourself:

146 kB: https://crates.io/crates/serde_json

43 kB: https://crates.io/crates/sj

29 kB: https://crates.io/crates/humphrey_json

7 kB: https://crates.io/crates/rust_json

kB of code doesn't quantify bloat.

most of the time those kB are documentation, which far outweigh any sort of bloatedness for a majority of people.

Reply on news.ycombinator.com