Reproducible Builds in January 2022
reproducible-builds.org
reproducible-builds.org
You can donate yourself. It is tax-deductible in US.
They include logos for donation larger than $25,000. Logos are here: https://reproducible-builds.org/who/sponsors/. Top sponsor is Google Open Source Security Team.
Of the entire set of contributors, most are paid zero. Some are paid.
Blaze (a.k.a. Bazel) rolled out at Google around 2007-ish, and the idea that consistent inputs should produce consistent outputs was fundamental to its philosophy.
Build rules like https://github.com/GoogleContainerTools/distroless to create minimal, reproducible docker images would seem radical and new most people building docker containers these days (almost everyone uses Dockerfile format) but it'd seem perfectly ordinary and very old-fashioned to any Googler.
Im trying to use it for game development, the model fits so perfectly for large monorepo projects with explicit dependencies, but it seems like since there’s no ecosystem that I have to write a lot of my own rules from scratch; and the learning resources are many years old and in the conference format. It’s quite dense to get in.
Even the slack requires an @google.com email to get in.
This shouldn't be the case, I'm in this slack and it's mostly non-google folks. Do you see this at https://slack.bazel.build ?
Which speaks to my larger point. Feels like there’s no strong community around this.
All I could find was invalid invite links and references to FaaS programs that would generate invites which were all failed.
This was about the time that Bazel was being open-sourced, and Matt's rules_docker extension was already in there. A solution existed, so to speak, but it would have been nutty to assume that the average project would switch from the straightforward-looking Dockerfile format to using Bazel and BUILD files to construct docker containers. And Docker Inc wasn't going to play along; they were riding a high valuation that depended on them being the final word about containerization, so vocally pretending the problem didn't exist was their safest way forward.
At one point I put together a process and POC for porting the concept of reproducible builds to docker in a user-friendly format -- essentially you'd define a spec that listed your dependencies with no more specificity than you needed. Then tooling would dep-solve that spec and freeze it into a fully-reproducible manifest that encoded all the timestamps, package versions, and other bits that would otherwise have been determined at build time. Then the _actual_ build process left nothing to chance: grab the identified sources and build and assemble in a hermetic environment. You'd attach the manifest to the container, and it gave you a precise bill of materials in a format that you could confidently use for identifying vulnerabilities. Since the builds were fully hermetic, a given manifest would only ever produce one set of bits, which could be reproduced in an automated fashion, allowing you to spot supply chain inconsistencies.
In my tooling, I leaned heavily on package providers like Debian as "owning" the upstream software dependency graph, since this was a problem they'd already solved, and Debian in particular was already serious about reproducibility in their packages.
In the end, it didn't go anywhere. There were a LOT of hacks to make it work since the existing software wasn't designed to allow this kind of integration. For example, the dependency resolution step required splicing in a lot of internal code from package managers, and and the docker container format was (and probably still is) a mess that didn't allow the end products to be properly identified as reproducible without breaking other things.
Plus, this is a problem that only people trying to do security at scale even care about. We needed a sea-change of industry thought around verifiability before my solution would seem at all valuable to people outside a few huge tech companies.
Funny to see you here. Matt and I haven't given up on this, we're giving a lot of that another try at Chainguard.
As I understand it, the goal should be that if you download a binary, and download the source and compile a binary, they should be identical.
I understand different build flags, and compilers. But if a project documents their build system why wouldnt it be reproducable?
Even without that, it's possible to accidentally leak entropy into the build output. For example, readdir() doesn't guarantee any kind of ordering, so without sorting the list of files it is possible for a binary artifact (or even tar) to produce different output from the same input.
For example if we just look at one of the listed reproducibility bugs in the report (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864082) you can get a flavor of the random bits and bods that need to be hunted down.
We could say that a system can be relatively easily made reproducible if starting from scratch - certainly all of the tooling is there. The far more significant challenge is retrofitting reproducibility into existing large projects.
Timestamps are often an implicit part of the build which leads to the same code producing 'different' artifacts (e.g. file created timestamp, generated metadata about the build stored in the artifact itself). One alternative is to support explicitly defining the 'build timestamp' so all builds with the same timestamp produces the same artifact. This is what Maven does[0]; I'd imagine it's much harder for build systems that download and recompile dependencies.
[0] https://maven.apache.org/guides/mini/guide-reproducible-buil...
Second of all, it's useful to have some sort of version marker that lets you tell at a glance if two copies of the software are the same, or what code was running when that log file was created. Build timestamps are a classic way of doing this in a version-control-system-independent way, without advancing the 'headline' version number for every developer's test builds.
And of course, if you include things like debug symbols it's useful to be able to tell if the source files on disk have changed since the build took place, as the line numbers could be all off.
Alternatives include using a git commit hash; having a separate build pipeline for public release versions then not worrying if developers' builds aren't reproducible; and just being careful with developers' test builds.
Imagine part of your build is generating a bunch of different files, and then packaging into an archive file, or into your installer bundle (or whatever). If your archive preserves file timestamps, then your end up with different archive files.
https://wiki.debian.org/ReproducibleBuilds/Howto#Files_in_da...
Notably, this affects archivers such as tar and zip, but there’s also a long tail of application-specific tools.
Maybe you want to distribute a .jar, .deb, .whl, or docker image?
Your question is kinda like "if I write a few hundred lines of code, and run it, it should run, right?". The answer is yes, that's the expectation, but there is no chance that it is actually fulfilled in practice.
As an example, when the Debian's reproducible builds project started, gcc had extra code to tag the executables with the time of compilation.
Perl sorting issues in dictionaries-common
libxmlb used a pointer address (%p) for a hash value
texlive-base: Reported differences in the generated ls-RI'm not saying you cannot build in parallel and end up with a deterministic build: that is done all the time. What I'm saying in that there are cases where some care has to be taken.
* Many file formats, especially (but not exclusively) archives like .zip, embed timestamps in them.
* Absolute paths are also ridiculously easy to get embedded in the build artifact.
* Other components of the user's environment can get embedded, including current locale, username, hostname. Some of these are more common than others.
* Using an actual random number generator is more common than you might think (e.g., generating a cryptographic key during the build process), although still pretty rare.
* Saying 'for file in directory' is usually sorted in inode order by default, which makes it inconsistent on different people's drives (although deterministic for a single user, generally).
* Pointer addresses are nondeterministic. And it's easy to accidentally make a container that sorts by pointer address (e.g., std::map in C++).
* In general, there's several features of programming languages that make it easy to accidentally inject nondeterminism in your program. Concurrency is obvious, but using standard containers like rust's HashMap or C++'s std::map combined with a "for every value in container" make it easy to get that nondeterminism. Compilers are not immune from being nondeterministic, and while it is a bug for them to be nondeterministic, it still happens.
(We discovered many surprising causes of nondeterminism during the development of Blaze aka Bazel, which prides itself on reproducibility.)
A lot of people have direct commit access and it is really hard to figure out near and medium term goals or their progress.
It's how the tool that does so much to ensure other things built are reproducible is not reproducible itself.
I am maintaining a small embedded linux project myself and having a reproducible build as a feature is a long standing goal. The more tooling and awareness there is, the better.