Hacker News Reader

I think build reproducibility is a cargo cult.

Most people here are debating you on the security angle, but in the case of Nix (and Guix) there is another important angle - reproducible builds make a content-addressed store possible.

In Nix, the store is traditionally addressed by the hash of the derivation (the recipe that builds the package). For example, lr96h... in the path

    /nix/store/lr96h3dlny8aiba9p3rmxcxfda0ijj08-coreutils-8.32

is the hash of the (normalized) derivation that was used to build coreutils. Since the derivation includes build inputs, either changing the derivation for coreutils itself or one of its inputs (dependencies) results in a different hash and a rebuild of coreutils.

This also means that if somebody changes the derivation of coreutils every package that depends on coreutils will be rebuilt, even if this change does not result in a different output path (compiled package).

This is being addressed by the new work on the content-addressed Nix store (although content-addressing aws already discussed in Eelco Dolstra's PhD thesis about Nix). In the content-addressed store, the hash in the path, such as the on above is a hash of the output path (the built package), rather than a hash of the normalized derivation. This means that if the derivation of coreutils is changed in such a way that it does not change the output path, none of the packages that depend on coreutils are rebuilt.

However, this only works reliably with reproducible builds, because if there is non-determinism in the build, how do you know whether a change in the output path is changed as a result of changing a derivation or as a result of uninteresting non-determinisms (the output hash would change in both cases).

Where the dependency chain is long, this substantially reduces build work during development too.

I'd guess that more than half of the invocations of gcc done by Make for example end up producing the exact same bit for bit output as some previous invocation.

I would point out that is literally what ccache (and Google goma) does, but doesn't require deterministic builds. Instead, it records hashes of preprocessed input and compiler commandlines.

They don't make any security claims about this, it's just for speeding up builds.

What we currently do --- hashing inputs --- is the same ccache way. We just don't yet sandbox with the granularity yet.

What we want to id hash outputs. Say I replace 1 + 2 with 0 + 3. That will cause ccache to rebuild. We don't want downstream stuff to also be rebuilt. C-linking withing a package is nice in parallelizable, but in the general case there is more dependency chains and now that sort of thing starts to matter.

I don't really have any complaints about using deterministic builds for non-security reasons, but the number one claim most proponents make is that it somehow prevents backdoors. Literally the first claim on reproducible-builds.org is that build determinism will prevent threats of violence and blackmail.

Another non-security angle: doesn't computer science also face a kind of replicability crisis related to the ability to acquire and compile source code associated with some published papers? Reproducible builds directly address that.

And it seems like even when that problem is resolved for the empirical component of computer science, bit-identical reproducibility could be valuable in case binaries are never submitted or distributed. This NixOS release is in a way a benchmark for how far we can currently get on a 'useful' system with that kind of reproducibility.

Honestly I think the biggest benefit of reproducibility is just debuggability. We both check out the same git repo and build it, we can later hash the binary and compare the hashes to know we're running the exact same code.

On security, if you really care about compromised build servers you might as well just build from source yourself. I think reproducibility might matter most in systems where side loading is hard/impossible like app stores, but I'm not familiar with the current state of the art in terms of iOS reproducable builds and checking them.

Reproducability is an option to mitigate backdoors and incentive developers to operate openly. It's no panacea, but it makes a lot of sense in open-source projects where individual actors are going to represent your largest threat vector. That way, it becomes a lot harder to push an infected blob to main, even if it still is technically possible. Hashes are also "technically pointless", but we still implement them liberally to quickly account for data integrity.

Signatures are not technically pointless, they mean you only have to trust the developer - not the mirror operators.

Reproducibility is technically pointless, because you still have to trust the developer, and they can still add backdoors.

Reproducibility means you don't have to worry that the developer might have a backdoored toolchain (which also means that they can't pretend that a malicious toolchain added the malicious code without their knowledge).

A talented developer might still be able to create a bugdoor which gets past code review, but that takes more effort and skill than just putting the malicious code into a local checkout and then saying "How did that get there?".

Every major vendor has vulnerabilities introduced all the time, by accident! No talent is necessary to introduce a bugdoor, just malice.

You can already verify that a toolchain wasn't backdoored today, reproducible builds aren't necessary for that.

> You can already verify that a > toolchain wasn't backdoored ) > today

How, exactly?

If we both compiled hello.c (a prototypical hello world program), and exchanged binaries; how would you verify my build wasn't malicious?

I think the workflow you're proposing is to take some trusted source code, then compile it to make a trusted binary. Now compare the trusted binary to the untrusted binary provided by the vendor - If they're the same - then it must have been made by an uncompromised toolchain.

That does require reproducible builds, but here is how to do it without reproducible builds:

Take the trusted source code, then compile it to make a trusted binary. Now put the untrusted binary in the trash, cause you already have a trusted binary :)

How about if the system will only run signed builds? Couldn't you use it to verify the signed build by stripping the signature and comparing them?

Is it technically pointless if you view it as a check on your own build, rather than a check on the work of others?

You are obviously familiar with Bazel/Blaze etc. Wouldn't reproducibility be necessary for those systems to work well most of the time? I can think of exceptions (like PGO), but it seems useful to produce at least some binaries this way. Also covered in this: https://security.googleblog.com/2021/06/introducing-slsa-end...

> Is it technically pointless if you view it as a check on your own build, rather than a check on the work of others?

That depends, I think it's difficult and mostly still pointless. I wrote about this a bit in the blog post I linked to. It's a big trade off, for questionable benefit.

> Wouldn't reproducibility be necessary for those systems to work well most of the time?

Yes, there are definitely some good non-security reasons to want deterministic builds. My gripe is only with the security arguments, like claims it can reduce threats of violence against developers (!?!).

> Reproducibility is technically pointless, because you still have to trust the developer, and they can still add backdoors.

Builder != developer - and with reproducible builds, you no longer beed to trust the builder. CI is commonly used for the final distributable builds and you can't always trust the CI server. Even if you do, many rely on third party thingd like docker images - if the base build image gets compromised, code could trivially be injected into builds running on it and without reproducible builds, that would not be detectable.

As a developer, it would be quite reassuring to build my binary (which I already do for testing) and compare the hash with the one from the CI server to confirm nothing has been tampered with. As a bonus, distro maintainers who have their own CI can also check against my hashes to verify their build systems aren't doing something fishy (malicious or otherwise).

> As a developer, it would be quite reassuring to build my binary (which I already do for testing) and compare the hash with the one from the CI server to confirm nothing has been tampered with.

That makes sense! However, this is not a good argument for reproducible builds, because you can already do that today.

You already have to build a trusted binary locally for testing right? You're dreaming of being able to compare that against the untrusted binary so that you can make sure it's a trusted binary too - but you already have a trusted binary!

Okay - but it's a hassle, you don't want to have to do that, right? Too bad - reproducible builds only work if someone reproduces them. You're still going to have to replicate it somewhere you trust, so you gained practically nothing.

With the reproducible build, you can start using the untrusted binary while you are still building your trusted one.

You can also have ten people on the internet verify the untrusted binary. With signatures, adding more people doesn't help.

> With the reproducible build, you can start using the untrusted binary while you are still building your trusted one.

That's not how it works, you have to reproduce it before it becomes trusted.

> You can also have ten people on the internet verify the untrusted binary.

Sure, then we have to build a complex consensus system that introduces a bunch of unsolved problems. My opinion is that this just isn't worth it, there is practically nothing to gain and it's really really hard.

> That's not how it works, you have to reproduce it before it becomes trusted.

Eh, there's stuff you can do with software before you trust it. Eg you can start pressing the CDs or distributing the data to your servers. Just don't execute it, yet.

> Sure, then we have to build a complex consensus system that introduces a bunch of unsolved problems. My opinion is that this just isn't worth it, there is practically nothing to gain and it's really really hard.

It's the same informal system that keeps eg debian or the Linux kernel secure currently:

People don't do kernel reviews themselves. They just use the official kernel, and when someone finds a bug (or spots otherwise bad code), they notify the community.

Similar with reproducible builds: most normal people will just use the builds from their distro's server, but independent people can do 'reviews' by running builds.

If ever a build doesn't reproduce, that'll be a loud failure. People will complain and investigate.

Reproducible builds in this scenario don't protect you from untrusted code upfront, but they make sure you'll know when you have been attacked.

> People don't do kernel reviews themselves. They just use the official kernel, and when someone finds a bug (or spots otherwise bad code), they notify the community.

There's a big difference here. When a vulnerability is found in the Linux kernel, that doesn't mean that you were compromised.

If a build was found to be malicious, then you definitely were compromised and it's little solace that it was discovered after the fact. This is why package managers check the deb/rpm signature before installing the software, not after.

Well, you'd still check the signature, to make sure you have the same build that the debian repository has.

This is just an additional check that the debian repository has sane builds.

(If someone mucks around with the debian repositories but you aren't the target, you might or might not be under attack.)

> What isn’t clear is what benefit the reproducibility provides. The only way to verify that the untrusted binary is bit-for-bit identical to the binary that would be produced by building the source code, is to produce your own trusted binary first and then compare it. At that point you already have a trusted binary you can use, so what value did reproducible builds provide?

That's not the interesting case. The interesting case is when the untrusted binary doesn't match the binary produced by building the source code. Assuming that the untrusted binary has been signed by its build system, you now have proof that the build system is misbehaving. And that proof can be distributed and reproduced by everyone else.

Once Debian is fully reproducible, I expect several organizations (universities, other Linux distribution vendors, governments, etc) to silently rebuild every single Debian package, and compare the result with the Debian binaries; if they find any mismatch, they can announce it publicly (with proof), so that the whole world (starting with the Debian project itself) will know that there's something wrong. This does not need any complicated consensus mechanism.

> More often, attackers want signing keys so they can sign their own binaries, steal proprietary source code, inject malicious code into source code tarballs, or malicious patches into source repositories.

In Debian, compromising the build server is not enough to inject malicious code into source code tarballs or patches, since the source code is also signed by the package maintainer. Unexpected changes on which maintainer signed the source code for a given package could be flagged as suspicious.

The only attack left from that list, at least for Debian, would be for the attacker to sign their own top-level Release file (on Debian, individual packages are not signed, instead a file containing the hash of a file containing the hash of the package is what is signed). But the attacker cannot distribute the resulting compromised packages to everyone, since those who rebuild and compare every package would notice it not matching the corresponding source code, and warn everyone else.

> I expect several organizations (universities, other Linux distribution vendors, governments, etc) to silently rebuild every single Debian package, and compare the result with the Debian binaries

This has been happening for many years. A lot of large companies that care about security and maintainability sign big contracts to with tech companies that often include indemnification.

There are other solutions to the problem of trusting maintainers; namely incremental distributed code review. The Rust folks are working on that:

https://github.com/crev-dev/

You still need Reproducible Builds and Bootstrappable Builds even if you have a fully reviewed codebase though.

>Developers have a perfect method for making their own software malicious: bugdoors.

I think rather than malicious developers the focus is on malicious build machines. How many things are built solely via CI these days, on machines that nobody has ever seen, using docker images that nobody has validated?

It's much easier to imagine a malicious provider (as in Sourceforge bundling in adware) than malicious developers, I think.

But yes, you're right that reproducible builds don't remove the need to trust the source.

>You have to trust the developer, but in theory, reproducible builds could be used to convince yourself their build server hasn't been hacked. This isn't really necessary or useful, you can already produce a trustworthy binary by just building the source code yourself.

This is pretty much all false though - not only the "just" part, as setting up a proper build environment is pretty non-trivial for many projects, and building everything from source is a task only the most dedicated Gentoomen would take up; you can also think of reproducible builds as a "litmus test". If you can, with reasonable accuracy, check whether a build machine is compromised at any time, you have a much greater base on which to trust it and its outputs. The benefits of having build machines probably shouldn't need explaining.

>You can't just pick one or two consensus servers, because then an attacker can stop you getting updates by compromising any one of them. You will have to do something like choose a lot of servers, and only require 51% to agree.

>...

>The problem is now you've either just replaced build servers with killswitches, or just replicated the same potentially-compromised buildserver.

I really don't understand this argument; compromised infrastructure probably shouldn't be a regular occurrence, and even if so, automated killswitches seem like the vastly more preferable option, no?

> I really don't understand this argument; compromised infrastructure probably shouldn't be a regular occurrence, and even if so, automated killswitches seem like the vastly more preferable option, no?

I'm pointing out how complex implementing reproducible builds is. It introduces a bunch of really hard unsolved problems that people are very handwavy about.

Who will do the reproducing? You say that users won't be able to do it. That makes sense, because if they could, then reproducible builds would be useless! However, you also say they will be able to check if a build server is compromised at any time. In order for both of those claims to be true we will have to design and build a complex consensus system operated by mutually untrusted volunteers. That's really hard, and seems like it provides a pretty negligible benefit.

You'll never beat the source.

IIUC Reproducible Builds guarantees that source is turned into an artifact in a consistent and unchanging way. So as long as the source doesn't change neither will the build.

I don't really understand what you're saying.

If you're saying "reproducible builds are reproducible", then that is obviously true, but the question is what is the benefit?

Some people claim that the benefit is that there will be less incentive to threaten developers with violence, and I'm saying that's nonsense. If you cut through the nonsense, there are some modest claims that are true, but doing reproducible builds properly is very complicated and the benefit is negligible.

> ... the question is what is the benefit?

I don't think I should have to explain this. It has nothing directly to do with violence against developers, that's taking many leaps.

It simply gives you what you expect, which is kinda the basis of safety and security.

> It has nothing directly to do with violence against developers, that's taking many leaps.

It is literally the very first claim on the front page of https://reproducible-builds.org.

That's just a bunch of marketing hype. I'm trying to stay focused closer to the matters at hand.

Perhaps my rambling on Development vs Distribution is relevant to the discussion? https://nixpulvis.com/ramblings/2021-02-02-signing-and-notar...

> If the server operators rebel, they can effectively veto a change the vendor wants to make.

How often do you think there will be a change so controversial that teams who have volunteered to secure the update system will start effectively carrying out a Denial of Service attack against all the users of that distro?

We also have to imagine that these malicious attestation nodes can easily be ignored by users just updating a config file, so the only thing the node operators could achieve by boycotting the attestation process is temporarily inconveniencing people who used to rely on them (which is not a great return on investment for the reputation they burn in doing this).

I don't know what reputation damage will happen, they're just third parties compiling code. There is no reputational damage for operating a malicious tor exit relay, why would this be different?

As I understand it, Tor does have a way of detecting whether an exit node is failing to connect users to their intended destination. (With TLS enforced, the only thing a malicious exit node could do is prevent valid connections).

In any case, I don't think anyone is proposing that the attestation nodes be run by random anonymous people on the internet. It would make more sense to have half a dozen or so teams running these nodes, with each team being known and trusted by the distro in question.

I'm not sure what the costs/requirements would be for running one of these nodes, but it might be possible for distros to each run a node dedicated to building each other's distros (or at least the packages that are pushed as security updates to stable releases).

Alternatively, individual developers that already work on a distro can offer to build packages on their own machines and contribute signed hashes to a log maintained by the distro itself.

The point was that a reproducible build doesn't mean you don't have to trust the developer.

Build servers rebelling was just an example of the additional complexities and attacks that it introduces, for very negligible benefit.