Hacker News Reader

Comment by dec0dedab0de | Hacker News Reader

Could someone with more knowledge explain why this is so difficult?

As I understand it, the goal should be that if you download a binary, and download the source and compile a binary, they should be identical.

I understand different build flags, and compilers. But if a project documents their build system why wouldnt it be reproducable?

View on HN

There are several things that can make builds nondeterministic:

* Many file formats, especially (but not exclusively) archives like .zip, embed timestamps in them.

* Absolute paths are also ridiculously easy to get embedded in the build artifact.

* Other components of the user's environment can get embedded, including current locale, username, hostname. Some of these are more common than others.

* Using an actual random number generator is more common than you might think (e.g., generating a cryptographic key during the build process), although still pretty rare.

* Saying 'for file in directory' is usually sorted in inode order by default, which makes it inconsistent on different people's drives (although deterministic for a single user, generally).

* Pointer addresses are nondeterministic. And it's easy to accidentally make a container that sorts by pointer address (e.g., std::map in C++).

* In general, there's several features of programming languages that make it easy to accidentally inject nondeterminism in your program. Concurrency is obvious, but using standard containers like rust's HashMap or C++'s std::map combined with a "for every value in container" make it easy to get that nondeterminism. Compilers are not immune from being nondeterministic, and while it is a bug for them to be nondeterministic, it still happens.

Great examples! Another one is the version number, which is really helpful in lots of situations. It can also be useful to embed the git commit for debugging. (One practical use case for these is cache busting to ensure that a browser downloads a new version of a script, for example.)

Another surprising example: by default, some compilers attempt more aggressive optimization if there is ample memory available.

(We discovered many surprising causes of nondeterminism during the development of Blaze aka Bazel, which prides itself on reproducibility.)

Depending on the tools used, it can be "easy" for things in the environment to leak into the build. Timestamps are a common bane of reproducible builds. I understand that a non-trivial amount of reproducible build work consists of hunting down every timestamp in a build, and then finding a way to inject alternate timestamps into them.

For example if we just look at one of the listed reproducibility bugs in the report (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864082) you can get a flavor of the random bits and bods that need to be hunted down.

We could say that a system can be relatively easily made reproducible if starting from scratch - certainly all of the tooling is there. The far more significant challenge is retrofitting reproducibility into existing large projects.

Why are people using timestamps in builds and what is the better alternative?

Some people want to know when something was built. Better option is to use Git commit of build or even last-modified of the file it's being embedded into. Alternatively, allow a env var to override what date/time the software thinks it is.

> Why are people using timestamps in builds and what is the better alternative?

Timestamps are often an implicit part of the build which leads to the same code producing 'different' artifacts (e.g. file created timestamp, generated metadata about the build stored in the artifact itself). One alternative is to support explicitly defining the 'build timestamp' so all builds with the same timestamp produces the same artifact. This is what Maven does[0]; I'd imagine it's much harder for build systems that download and recompile dependencies.

[0] https://maven.apache.org/guides/mini/guide-reproducible-buil...

First of all, every filesystem has timestamps and they've got to be set to some value.

Second of all, it's useful to have some sort of version marker that lets you tell at a glance if two copies of the software are the same, or what code was running when that log file was created. Build timestamps are a classic way of doing this in a version-control-system-independent way, without advancing the 'headline' version number for every developer's test builds.

And of course, if you include things like debug symbols it's useful to be able to tell if the source files on disk have changed since the build took place, as the line numbers could be all off.

Alternatives include using a git commit hash; having a separate build pipeline for public release versions then not worrying if developers' builds aren't reproducible; and just being careful with developers' test builds.

but why should different timestamps on the filesystem cause different binary?

Don't think binaries, think build outputs.

Imagine part of your build is generating a bunch of different files, and then packaging into an archive file, or into your installer bundle (or whatever). If your archive preserves file timestamps, then your end up with different archive files.

There’s a long list of tools that follow a general pattern: they take an input file and produce an output file, but the output file doesn’t just depend on the contents of the input file; it also embeds its modification time somewhere. Sometimes the goal is to be able to automatically regenerate the output if the input changes, by comparing timestamps. Sometimes it’s just intended as general-purpose metadata. Debian’s reproducible builds wiki page has many examples:

https://wiki.debian.org/ReproducibleBuilds/Howto#Files_in_da...

Notably, this affects archivers such as tar and zip, but there’s also a long tail of application-specific tools.

OP misspoke; the timestamp in the filesystem metadata doesn't matter for this. This is only about timestamps embedded in the binary itself.

Plenty of build outputs are compressed collections of files.

Maybe you want to distribute a .jar, .deb, .whl, or docker image?

By default, many compilers include things like local filesystem paths, build server hostnames, or build timestamps into their binary artifacts. These will obviously differ build-to-build.

Even without that, it's possible to accidentally leak entropy into the build output. For example, readdir() doesn't guarantee any kind of ordering, so without sorting the list of files it is possible for a binary artifact (or even tar) to produce different output from the same input.

Compilers are huge pieces of code, modified by lots and lots of hands, and up to recently nobody paid any attention if the results were identical or not.

Your question is kinda like "if I write a few hundred lines of code, and run it, it should run, right?". The answer is yes, that's the expectation, but there is no chance that it is actually fulfilled in practice.

As an example, when the Debian's reproducible builds project started, gcc had extra code to tag the executables with the time of compilation.

https://lists.reproducible-builds.org/pipermail/rb-general/2... should give you a better idea of what's going on. It's linked in the article.

  Perl sorting issues in dictionaries-common
  libxmlb used a pointer address (%p) for a hash value
  texlive-base: Reported differences in the generated ls-R

In addition to various issues already mentioned, there are also some build systems that do build in parallel and produce builds or build artefacts that are going to depend on the order in which threads finished, which may vary from run to run.

I'm not saying you cannot build in parallel and end up with a deterministic build: that is done all the time. What I'm saying in that there are cases where some care has to be taken.

The build environment seems to sneak into the artifacts in many ways. Check out some of the listings from the "Upstream Patches" section of the linked page.. Some of the most common reasons seem to be: timestamps, non-deterministic file listings, debug information containing build path data, permissions bits on artifact files,etc.

Apart from what others have said, signed binaries also pose somewhat of a challenge with reproducible builds.

Reply on news.ycombinator.com