Is my package reproducible yet?
ismypackagereproducibleyet.org
ismypackagereproducibleyet.org
I'm not 100% certain, but I believe this is an area where Golang is doing better (code references a Git repo directly).
Are there any other languages that focus on "reproducible builds" for packages? (Ideally via Git)
Edit: I just want to add that this is a major frustration for me with Java too. Everything on Maven barely has any metadata to go back and audit the source code!
In particular, Maven's use of groupId:artifactId:version is inadequate as an identifier, since many artifacts can have this same "identifier", e.g.
- Making changes without updating the version number (bad practice, but easily done)
- Building with different versions of dependencies. Maven allows version ranges, and resolves them based on the contents of third-party servers, so this can change over time even if we're building from the same git commit.
- Building with different property settings, e.g. due to a different profile getting 'activated' due to some ambient aspect of the system (OS, presence/absence of some file path, env vars, etc.).
- Building with different versions of Maven, Java, etc.
- The contents of Maven's cache also affects builds. If a file path exists in the cache, Maven will use it regardless of its contents. Also, undeclared dependencies can sometimes be picked up from the cache, which would otherwise fail if they weren't in the cache.
Maven also uses that scheme as a location for fetching artifacts, which is not only inadequate but also insecure (we can put arbitrary code at a certain path and Maven will happily fetch it). Maven does perform SHA512 verification, but those hashes are fetched from the same location as the artifacts, so subverting one allows subverting both.
Also, in case you weren't aware, git tags are just a convenience. Commit IDs are SHA1 hashes, so they're unforgeable and tamper-proof; whereas git tags can be deleted and recreated to point at anything.
FYI at my workplace we avoid some of these issues by:
- Performing all builds using Nix, which uses a sandbox with versions of Maven, JVM, etc. specified with their SHA256 hashes. This avoids differences due to tooling, profiles, env vars, caches, etc.
- All builds are performed without network access. This prevents Maven resolving version numbers differently.
- The SHA256 of the build environment (including all tooling, env vars, properties, source code, etc.) gets appended to the version number, which distinguishes these different artifacts from each other.
- Our projects specify the expected hash of their Maven dependencies (essentially like a lock file), which avoids having to trust third-parties for the hashes (a trust-on-first-use policy).
Yes, the situation's pretty bad. On the plus side, I'm increasingly of the opinion that "package management" is unnecessary; and usually just a symptom of having a poor build system.
That's not what reproducible means in this context. Being able to go from a given release artifact to a git repo is not what the "reproducible builds" project/movement/whatever is about.
Reproducible builds are more about the packaging of something than about specific language ecosystems, so asking about languages that focus on reproducible builds makes less sense than asking about package ecosystems, build systems, and linux distros that do.
Let's look at an example: docker is written in go, and is marked as reproducible in debian, but unreproducible in archlinux: https://ismypackagereproducibleyet.org/?pkg=docker&query=que... (named docker.io on debian https://ismypackagereproducibleyet.org/?pkg=docker.io&query=...)
Why is that?
Well, debian sets the "version.BuildTime" option when it compiles it to a constant value each time it compiles it: https://salsa.debian.org/go-team/packages/docker/-/blob/4f1f...
Archlinux's PKGBUILD doesn't https://github.com/archlinux/svntogit-community/blob/cf04bef...
Because of that, "docker version" on arch linux outputs that it was built on the actual day it was built, whenever that was, and each time it's built that string changes. Hence, not reproducible.
Reproducible builds are about capturing enough information about how something is built, and specific enough build instructions, that two people can produce the same bit-for-bit identical output, and therefore verify that someone didn't compile a malicious backdoored binary, and also increase the chance that each build consistently works or doesn't work, regardless of who compiles it when.
I feel like I understand why this is done, but it still feels extremely odd. We're making tools and such lie to us in the pursuit of reproducibility and oftentimes this means erasing certain information, that we might be interested in actually keeping!
Surely, we should be able to do better than that, something like:
/my-package
/reproducible
[CONSISTENT_BUILD_OUTPUT_FOR_CODE_INPUT]
/non-reproducible
build-time.txt
build-system-information.txt
build-log-with-timestamps.txt
build-package-mirrors-that-were-contacted.txt
build-step-timing-information-and-traces.txt
[OTHER_STUFF_GOES_HERE]
That said, it's absolutely annoying to build a container image with Jib (just an example from when I last tried) and then see in Docker that its build time was supposedly decades ago. While having incorrect timestamps isn't the worst thing ever, I can imagine other pieces of information mattering more.
Fwiw, I'm the maintainer of Docker on Arch and this is a bit more complicated.
Arch doesn't need to care about build time because `pacman` sets `SOURCE_DATE_EPOCH` during package building which is then utilized by the docker build system.
https://github.com/moby/moby/commit/760763e9957840f1983a5006...
If you look at the diff from an actual reproduction of the `docker` package (the Debian Reproducible Builds CI is a fuzzer), you will see there is no issues around build time.
https://reproducible.archlinux.org/api/v0/builds/359851/diff...
What you do see is some weird differences around `NT_GNU_BUILD_ID` and `GO BUILDID`.
This is because of `lto`. In Arch we try to build most Go packages with cgo for the purpose of utilizing hardening flags. The issue is that the C code generation in Golang isn't actually reproducible with `lto` enabled.
One issue is what I submitted here; https://github.com/golang/go/pull/53528
Another issue which I have been trying to debug is why the code snippet located here doesn't reproduce.
This results in the BUILDID generated and sat by the cgo build process isn't reproducible.
I could disable lto and all the binary hardening and docker would probably be reproducible, but where is the fun in that :)?
Sorry for being so off the mark there for your package, though my incorrect understanding I think did allow for a good explanation of what reproducibility is about and one thing that commonly fails!
Those linux distributions often include their own custom build scripts and patches to make things reproducible. Debian isn't just execing "npm install" or "cargo install" when you "apt install <package>", so "reproducibility" in the context of this webpage doesn't necessarily mean anything at all about specific crates/packages, other than that the debian/arch/opensuse maintainers were able to make their build process for that package, possibly with patches, reproducible.
Some distros actually do run `cargo install` or similar during build, although Debian does not indeed.
Sure, _updating_ the crate might be a nightmare if the generation part is not stable - but that's unrelated to reproducibility.
Since code is generated, that could be considered the same as saying "you know you're getting the same binary every time" for pre-compiled binaries. The key argument is whether "generated code" can be considered "source code" or not.
Personally, I'd say no - "source code" almost always refers to the preferred representation that humans are editing. If code is generated, then the true source code is the code of the generator and its input. A similar distinction is declared in the GNU General Public License, to prevent companies from circumventing the spirit of the license by releasing obfuscated source code.
Rule of thumb: If the original author is not editing it directly, it's not source code.
Besides, it's not a problem that Cargo alone can solve. At the end of the day it's picking up code from a git repo, and how you produced the code in that repo is not something it could control.
I think you misunderstand what "reproducible" means in the context of OP, but I will try to answer your question in the context of Rust.
When you upload a crate to crates.io it gets uniquely identified by its cryptographic hash. It's not possible to replace code for an already uploaded crate version. Now, when you get dependencies for your build you may get newer (semver-compatible) version, but you always can retrieve the older version. To prevent such automatic updates, Rust applications often commit Cargo.lock to their repos, which contains hashes of all dependencies. So unless you manually run `cargo update` you will get exactly the same dependency tree, which with some minor build tweaks makes properly reproducible builds possible.
Arguably, it's a much better system than relying on VCS as Go does.
Yes, it's (IMO unfortunately) possible to have non-reproducible code generation during compile time. The most prominent example is code generation based on DB schema. But such cases are quite rare and "going with the VCS" does not help here in any way.
If you mean that Rust code gets generated by a Python script and then gets uploaded to crates.io without the "origin" scrips, then I don't see any problems here.
>Or crates where the tests were in VCS but not in the crate
Tests are not used in building the final application, so they are irrelevant for reproducibility.
>prebuilt files in the crate
While I think it's a bad practice (though I haven't seen such crates myself), it does not hinder build reproducibility.
It does, because then you can run the build process yourself from the real source files and check if it is reproducible.
> If you mean that Rust code gets generated by a Python script and then gets uploaded to crates.io without the "origin" scrips, then I don't see any problems here.
Right. The problem is there is no source in the crate, just generated Rust code, so you can't know if the build process to create that Rust code is reproducible or not.
> While I think it's a bad practice (though I haven't seen such crates myself), it does not hinder build reproducibility.
The problem is that these are prebuilt files and so your build process is just copying them not building them, so you can't know if the build of them is reproducible.
It would be good to flag bad packages but it's impossible.
Also, releases on maven central typically include a sources.jar along with the binaries. It's required to provide that as well as javadoc to publish on maven central, I think. They also require jars to be signed and there are some manual checks before they activate your account to verify e.g. domain ownership and metadata of your project. It's not perfect but it's better than what many other packagemanagers do. It's actually a bit of a PITA to setup; I've wasted quite a bit of time getting some stuff published there just trying to figure out their convoluted processes and tools and error messages. This stuff is way too hard in it's current form.
Not all projects that use these build tools are fully reproducible but that is probably pretty easy to fix if people raise awareness of issues around this topic.
There's even a website that tracks reproducibility for some common libraries on maven central: https://github.com/jvm-repo-rebuild/reproducible-central#rea...
If you want to use artififacts produced straight from git hashes, tags, or branches, jitpack.io is pretty neat.
Packages are downloaded directly from GitHub or whatever repo configured.
NPM isn't a language. (Nor is NodeJS, for that matter.) It doesn't even occupy some semi-official position in its relationship to the actual language. Often, the NPM/NodeJS way even stands in stark contrast—working against the spirit and sometimes even the letter of the relevant standards and language features.
(It is pretty remarkable, I admit, how the opinionated set of practices associated with NPM/NodeJS have managed to reach the place it has in the associated cultural dominance hierarchy, to the point that people often forget about this stuff.)
> (Ideally via Git)
Yeah, and it's super easy: you can make your builds reproducible by committing your source code to Git. All of it. (I.e., the libraries you're using, too.) People get irrationally upset when you suggest it, though; it really feels like it shouldn't be that easy, so they don't let it.
I have a suspicion that you're not putting the source for the specific versions of glibc and Linux you used into every one of your projects.
I don't produce anything where Linux's relationship to the application is as something that lives inside the build (unlike the types of third-party NPM dependencies we're talking about here) and I have a suspicion that there isn't any Linux inside the artifacts that your build step produces, either.
If that's not the case for you, then yeah, I'll bite the bullet. You absolutely can either: (a) include that source code in your repo or (b) stop building things like that. If you find that going with the former option isn't easy, then that's because it never was—whether you're checking in all the source code or not.
Fair point! I should have said "Are there any other package managers that focus on reproducible builds?" (Or maybe "ecosystem" would be a better term?)
> you can make your builds reproducible by committing your source code to Git.
That doesn't help when, for example, a TypeScript NPM module is compiled and "released" with only the JavaScript code + `*.d.ts` files. Or, worse, sometimes the release is entirely minified. (I feel like Deno might be tackling this problem, but I'm not sure.)
I agree that checking in the dependencies helps (we do that via Yarn PnP, which has it's own pain points too). It's still not the "end game" solution though like what Nix is targeting.
I don't understand. How does it not help in that scenario (or the minification one)?
If a `make install` invocation gzips manpages instead of the package manager or distro hooks, then you won't have reproducible manpages unless the upstream uses the `-n` switch as well. This is just an example of many small and obscure issues that hide inn different upstream projects.
It did a fantastic job of highlighting the few changes in a tree of XML "one liner" files (each file one ~ 25 Kb text line).
https://reproducible-builds.org/citests/ https://r13y.com/ https://data.guix.gnu.org/repository/1/branch/master/latest-...
We hover around 85% and 89% reproducible packages across all our packages and no distro, to my knowledge, is close to anything like this.
Arch Linux has a rebuilder setup and you can safely reproduce between 85% and 89% of the repository packages.
Nix has never done such a thing, and would be far away from any comparable numbers.
Nix is 99.77% reproducible, so come on. It’s not marketing, nix just properly solves the problem for the first time ever.
Guix is another example, it is more into “bootstrappability” though, so packages can be built from source all the way down to a few kilobyte C-subset compiler written in assembly. Very cool!
You haven't read r13y properly.
>1733 out of 1737 (99.77%) paths in the nixos.iso_minimal.x86_64-linux installation image are reproducible!
600 packages becomes 1737 paths which are the inputs used to generate the image. Several of the compared paths are just static files which doesn't change, so the "path to package" comparison doesn't work either.
EDIT: An example set of stuff it attempts to compare can be found here:
https://paste.xinu.at/TmVM0y5FBDWX/
Taken from the build here: https://buildkite.com/grahamc/r13y-dot-com/builds/1149#01813... They have not solved the problem.
Arch has spent a considerably amount of time getting to the figure it has and is in the same position as NixOS to maintain reproducible builds. The claim that NixOS solves this, or is uniquely dispositioned to solve this, is the false. That is what I'm effectively trying to point out here.
>While it is a different problem, my litmus test for package managers is installing gnome a (the whole group) and then seeing whether I can transition to kde or vice versa without a bunch of dependencies lingering around forever. Pacman, apt, etc. all fail terribly here.
That is not what Reproducible Builds is about.
I do know it is not about reproducible builds, I wrote as much, but I still feel that dismissing nix as you did above is not professional.
For example: https://tests.reproducible-builds.org/debian/rb-pkg/unstable...
If so, how would you imagine that could be sped up?
- If the distro is working on reproducibility, they should have a dashboard somewhere with reproducibility status.
For arch that would be https://reproducible.archlinux.org/
It would be done in their CI and there should be no reason for you do also manually verify this, unless you are a very targeted individual and you think its reasonable that arch linux infrastructure is hacked to target you.
Basically, either you trust your distro or you don't. If you are running local verifications of reproducibility, I am guessing you are not.
So consider switching to a distro that you do trust.