Lots of progress for Debian's reproducible builds
lwn.net
lwn.net
Offhand, I remember that /etc/{passwd,group} are copied from the host machine by design. There is also a random seed file, to save entropy across reboots. And there is some nondeterminism in the dynamic linker cache AFAIK. And timestamps in logs.
If anyone is interested in this let me know.
etc/resolv.conf
var/cache/ldconfig/aux-cache
var/lib/urandom/random-seed
etc/init.d/.depend.{stop,start,boot}
etc/shadow
etc/passwd- and family (with trailing hyphen)
etc/apt/trustdb.gpg and other keys
I have a cron job running daily on multiple machines, doing a deterministic debootstrap of Debian Wheezy on i386 and amd64.
It basically wraps debootstrap, strips down the image a bit, and stamps out the nondeterminism. Every day it has gives a checksum for i386 and one for amd64, which holds across multiple host machines (one is Debian Wheezy, the other is Ubuntu Trusty). So it is free from host influence.
The /etc/apt/sources.list just has the "wheezy" repo now (i.e. not wheezy-updates). So Debian 7.7 had one pair of checksums, and then on Jan 10 2015, I noticed Debian 7.8 was released. They changed that day, and have been stable/reproducible every day since.
Part of this is also mirroring the Release/Packages metadata daily and storing version history in Git. One nice thing I found out about Debian through doing this is that the Release file completely describes the input, since it's hashes all the way down (a "Merkle tree", basically like Git itself.) My scripts also make it so you can store versioned metadata in one tree, while keeping data immutable in "pool" (all this really requires is symlinks and file:// URLs for the repo).
What project were you working with in this area? I'm basically doing this to make reproducible builds of containers. I'm kind of surprised that Docker completely punts on this problem.
I've found myself automating Debian/kernel/boot loader/libs/app stack packages for some ARM hardware for which I'm an app developer. This is due to the outrageous situation of COM module vendors (and/or SoC manufacturers) thinking that kernel 3.0 and horribly ancient/unsupported/unpatched userlands are acceptable in 2014/2015.
I run our .deb builds (and final debootstrap) in a bunch of docker containers, given that xapt/debcross toolchain has some stuff that, although these are awesome in themselves, still have some quirks when building stuff with awkward/complex dependencies and require a lot of hand-holding normally. So dockerizing the build env makes my life easier (some of it still has to be done from arm native chroot via qemu+binfmts). I use Jenkins to automate and integrate with the rest of our build tools but make files work too.
The /etc/passed et. al. copying from host is definitely the wrong thing to do in my circumstances as well; it's curious the various failure modes chroot builds can have.
https://github.com/andychu/debian-wheezy-metadata
(NOTE: The repo takes up 300+ MB on my local disk)
I chatted on #debian IRC about this a little. It seemed like one person thought doing all the archs would take up too much space. So far doing it daily for over a year for two archs has been manageable. I think you simply have one repo per arch, instead of having multiple archs in one repo like I have here.
I will try to get the code up... if you think you will use it you can ping me on github. I think this should be in Debian itself, and actually supported by debootstrap, but so far I am just wrapping it rather than patching it. (I started off trying to patch it.)
It's an interesting use-case for Docker as well: you can ship the build environment (or its Dockerfile describing it) for people to run builds under the same env as the official released build.
Many many build processes have hidden from-the-network dependencies which are yet another huge source of problems (both in terms of having high-availability for the build process and in terms of understanding exactly what is getting built).
All of the upstream Docker images (including the bottom base images which most people never look at carefully) would need to be built in a reproducible way. And the Docker code would need to actually verify checksums more carefully than it currently does.
Having projects like Debian doing work like this will make it a lot easier for everything else to become reproducible. Which in addition to the security benefits is also pretty useful when bug hunting - having a system that makes it clear which (if any) of your dependencies changed narrows the list of things which need to be checked when something breaks. Ubuntu Core, Snappy, and Nix are also on my list of things to watch in this area.
NixOS is definitely on my list, although for different reasons (whilst I love using and babysitting Debian systems, orchestrating it is eating up time that could be better spent on something like NixOS).
If these commits were used to adjust version numbers in source files, the following trick should eliminate them: Check out the release branch into a working copy, adjust the version numbers in the working copy, and then copy the working copy to the tag's URL (as in: cd working-copy; svn copy . ^/tags/1.0)
If this was a maven project: The maven release plugin is doing this wrong and performs 3 commits to an SVN repository per release...
I didn't mean to criticize SVN as being inherently incapable of reproducible builds (if anything that's harder to achieve with git, especially with hacks like I've listed above), but the act of cleaning up the SVN repos and preparing for migration to git where a lot of our old SVN (and RCS!) habits would be problematic, also seem like the same kind of housecleaning you'd need to prepare for reproducible builds.
I mean, if a package is compiled on the same system, with the same compiler, with the same build script -- should it not produce the same output?
Other problems include non-deterministic filesystem order, randomized hash algorithms, and even the fact that Markdown processors mangle email addresses randomly. A highly vexing problem that I'm currently trying to solve is that libxslt implements the XSLT generate-id() function by taking the memory address of the XML node struct, which makes documentation generated with XSLT non-reproducible (memory addresses are non-deterministic because of address space randomization and randomized hash tables).
(I'm the author of strip-nondeterminism, a tool in our custom toolchain that attempts to normalize files after they're built.)
Browse source @ https://anonscm.debian.org/cgit/reproducible/strip-nondeterm...
PS. non-deterministic filesystem order .. doesn't this go away if you use tmpfs?
My current solution is a patch that uses a hash table to map the memory address of a node to a counter that increments in a deterministic order. It's a massive hack and I hate it. I'm currently trying to assess what its performance impact is.
The proper solution would be for libxml2 to maintain a counter and assign every node a deterministic ID, but that would require extending the _xmlNode struct, which would break ABI compatibility because the struct is not opaque.
I've been wondering -- why do compiled binaries actually include timestamps? For human-readable stuff, I sort of understand, but for binaries I don't see we'd want to include timestamps.
Also, the goal is to make it reproducible across machines, since one of the motivations is to enable everybody to verify that the binary distribution matches the source.
You can blame the compiler random number seed for much non-determinism. I googled for this and found at good explanation at:
http://blog.mindfab.net/2013/12/on-way-to-deterministic-bina...
The summary is anonymous namespaces actually have a name, that being a big integer, and they're all distinct so each has a different one and gcc just picks a psuedorandom number for each anon namespace. They're anonymous to you, but not to the compiler LOL crazy but true. Some folklore that once every 50 trillion years (maybe less often) linking two compiled files will fail at runtime not compile time because two namespaces randomly were assigned the same number so they "crossed the streams".
Supposedly everyone knows some versions of GCC sometimes pick specific optimizations pseudorandomly. This is one of those "everyone knows and nobody has evidence" things. I'd welcome a link to actual evidence, like to actual code that sometimes compiles differently based on phase of moon or whatever. If I were more bored I'd make a github project specifically to manipulate gcc for fun, sounds amusing.
If you're compiling up static libraries, AR is just a generic, although crude and ancient, file archiver and therefore contains timestamps as explained in the link below. I don't think that's necessary for static library operation its just an artifact of the file format.
http://en.wikipedia.org/wiki/Ar_%28Unix%29
There is also a funny failure mode where two systems with somewhat different libraries (perhaps one upgraded after the other, or a security patch, or got owned), when compiling a statically linked binary with the same code and compiler will obviously have different static linked result. This is of the same class as optimizations getting a little too personal for your specific CPU family such that two generic amd64 intel boxes are not quite so identical as you'd think in both optimizations and runtimes. (oh edited to add what if your virtualization is funky such that GCC didn't think a certain instruction set was present in the virtual image, so bare metal compiles would have different optimizations than images running on the same bare metal, although this is totally theoretical and probably doesn't exist in reality)
Also openSUSE have reproducible builds/packages since ages via OBS (http://build.opensuse.org) and now Factory/Tumbleweed have reproducible packages + automatic CI (using openQA: https://openqa.opensuse.org) Quite an achievement for a rolling distribution.