Hacker News Reader

Comment by nimish | Hacker News Reader

As a user of fedora what does this actually get me? I mean I understand it for hermetic builds but why?

Reproducible builds can improve software quality.

If we believe we have a reproducible build, that's constitutes a big test case which gives us confidence in the determininism of the whole software stack.

To validate that test case, we actually have to repeat the build a number of times.

If we spot a difference, something is wrong.

For instance, suppose that a compiler being used has a bug whereby it is relying on the value of an unitialized variable somewhere. That could show up as a difference in the code it generates.

Without reproducible builds, of course there are always differences in the results of a build: we cannot use repeated builds to discover that something is wrong.

(People do diffs between irreproducible builds anyway. For instance, disassemble the old and new binaries, and do a textual diff, validating that only some expected changes are present, like string literals that have embedded build dates. If you have reproducible builds, you don't have to do that kind of thing to detect a change.

Reproducible builds will strengthen the toolchains and surrounding utilities. They will flush out instabilities in build systems, like parallel Makefiles with race conditions, or indeterminate orders of object files going into a link job, etc.

I don't think it is that unlikely that build hosts or some related part of the infrastructure gets compromised.

I don't know this area, but it seems to me it might be a boon to security? So that you can tell if components have been tampered with?

Bingo. We caught a virus tampering with one of our code gens this way.

Yes! The attack on SolarWinds Orion was an attack on its build process. A verified reproducible build would have detected the subversion, because the builds would not have matched (unless the attackers managed to detect and break into all the build processes).

Exact same binary can be hacked in exact same way on all platforms.

What?

If I understand correctly that would require releasing the publisher’s private key, though，correct?

Only if you try to reproduce the signature. Usually the signature is stored separately. That way, the reproduced work's signature applies to it as well.

That would requiring breaking each of the separate build processes, which is very unlikely.

This doesn't counter subverted source code, that's not what reproduciblrle builds are for.

for related, see Ken Thompson:

http://genius.cat-v.org/ken-thompson/texts/trusting-trust/

That's already been a thing in all the Redhat variants. RPM/DNF have checksums of the installed binaries and there is GPG signing of packages and repositories. The only part of that ecosystem I've always had a gripe with is putting the GPG public keys in the mirrors. People should have to grab those from non mirrors or any low skilled attacker can just replace the keys and sign everything again. It would be caught but not right away.

Changes can also be caught using bolt on tools like Tripwire, OSSEC and it's alternatives or even home grown tools that build signed manifests of approved packages usually for production approval.

You know what does not give me confidence? Updating software, but whats that, its still printing the same build date? Of course hours later tens of files deep I found out some reproducability goof just hardcoded it.

So far, reproducible builds are heavy on the former, zero on these bugs you mention and zero on supply chain attacks.

Some of these supposed use cases make no sense. You update the compiler. Oh no, all the code is different? Enjoy the 16h deep dive to realize someone tweaked code generation based on the cycle times given on page 7893 of the Intel x64 architecture reference manual.

They should be setting the build days for a package from say the commit date of the top commit of the branch that's being built. It can't be something that doesn't change when the next version is spun. If you see a behavior like that in anybody's reproducible package system or distro, you have a valid complaint.

My impression is that reproducible builds improve your security by helping make it more obvious that packages haven't been tampered with in late stages of the build system.

* Edit, it's quoted in the linked article:

> Jędrzejewski-Szmek said that one of the benefits of reproducible builds was to help detect and mitigate any kind of supply-chain attack on Fedora's builders and allow others to perform independent verification that the package sources match the binaries that are delivered by Fedora.

Bingo.

The supply chain attacks you have to most worry about are not someone breaking into Fedora build machines.

It's the attacks on the upstream packages themselves.

Reproducible builds would absolutely not catch a situation like the XZ package being compromised a year ago, due to the project merging a contribution from a malicious actor.

A downstream package system or OS distro will just take that malicious update and spin it into a beautifully reproducing build.

Don't let the perfect be the enemy of the good; this doesn't prevent upstream problems but it removes one place for compromises to happen.

I'm not saying don't have reproducible builds; it's just that this is an unimportant justification for them, almost unnecessary.

Reproducible builds are such an overhelmingly good and obvious thing, that build farm security is just a footnote.

Your mere footnote is my soft, soft underbelly.

Any hardening is still hardening.

And anything designed to catch upstream problems like the XZ compromise will not detect a compromise in the Fedora package build environment. Kinda need both.

Reproducible builds COULD fix the xz issues. The current level would not, but github could do things to make creating the downloadable packages scrip table and thus reproducible. Fedora could checkout the git hash instead of downloading the provided tarball and again get reproducible builds that bypass this.

The above are things worth looking at doing.

However I'm not sure what you can code that tries to obscure the issues while looking good.

When builds are reproducible, one thing a distro can do is have multiple build farms with completely different operators, so there's no shared access and no shared secrets. Then the results of builds of each package on each farm can be compared, and if they differ, you can suspect tampering.

So it could help you detect tampering earlier, and maybe even prevent it from propagating depending on what else is done.

Better security! A malicious actor only needs to change a few bytes in either the source or binary of OpenSSL to break it entirely (i.e. disable certificate checking).

Reproducible builds remove a single point of failure for authenticating binaries – now anyone can do it, not just the person with the private keys.

It's one tool of many that can be used to prevent malicious software from sneaking in to the supply chain.

Keep in mind that compilers can be backdoored to install malicious code. Bitwise/signature equivalency does not imply malware-free software.

True, but every step we add makes the others harder too. It is unlikely Ken Thompson's "trusting trust" compiler would detect modern gcc, much less successfully introduce the backdoor. Even if you start with a compromised gcc of that type there is a good chance that after a few years it would be caught when the latest gcc fails to build anymore for someone with the compromised compiler. (now add clang and people using that...)

We may never reach perfection, but the more steps we make in that direction the more likely it is we reach a point where we are impossible to compromise in the real world.

In this attack, the compiler is not a reproducible artifact? Or does backdooring use another technique?

Their point is that you'd need something like https://bootstrappable.org/ (which does exist).

Reply on news.ycombinator.com