Comment by jchw | Hacker News Reader

Comment by jchw | Hacker News Reader

I think this debate comes down to exactly what "reproducible" means. Nix doesn't give bit-exact reproducibility, but it does give reproducible environments, by ensuring that the inputs are always bit-exact. It is closer to being fully reproducible than most other build systems (including Bazel) -- but because it can only reasonably ensure that the inputs are exact, it's still necessary for the build processes themselves to be fully deterministic to get end-to-end bit-exactness.

Nix on its own doesn't fully resolve supply chain concerns about binaries, but it can provide answers to a myriad of other problems. I think most people like Nix reproducibility, and it is marketed as such, for the sake of development: life is much easier when you know for sure you have the exact same version of each dependency, in the exact same configuration. A build on one machine may not be bit-exact to a build on another machine, but it will be exactly the same source code all the way down.

The quest to get every build process to be deterministic is definitely a bigger problem and it will never be solved for all of Nixpkgs. NixOS does have a reproducibility project[1], and some non-trivial amount of NixOS actually is properly reproducible, but the observation that Nixpkgs is too vast is definitely spot-on, especially because in most cases the real issues lie upstream. (and carrying patches for reproducibility is possible, but it adds even more maintainer burden.)

[1]: https://reproducible.nixos.org/

> The quest to get every build process to be deterministic [...] will never be solved for all of Nixpkgs.

Not least because of unfree and/or binary-blob packages that can't be reproducible because they don't even build anything. As much as Guix' strict FOSS and build-from-source policy can be an annoyance, it is a necessary precondition to achieve full reproducibility from source, i.e. the full-source bootstrap.

Nixpkgs provides license[1] and source provenance[2] information. For legal reasons, Nix also defaults to not evaluating unfree packages. Not packaging them at all, though, doesn't seem useful from any technical standpoint; I think that is purely ideological.

In any case, it's all a bit imperfect anyway, since it's from the perspective of the package manager, which can't be absolutely sure there's no blobs. Anyone who follows Linux-libre releases can see how hard it really is to find all of those needles in the haystack. (And yeah, it would be fantastic if we could have machines with zero unfree code and no blobs, but the majority of computers sold today can't meaningfully operate like that.)

I actually believe there's plenty of value in the builds still being reproducible even when blobs are present: you can still verify that the supply chain is not compromised outside of the blobs. For practical reasons, most users will need to stick to limiting the amount of blobs rather than fully eliminating them.

[1]: https://nixos.org/manual/nixpkgs/stable/#sec-meta-license

[2]: https://nixos.org/manual/nixpkgs/stable/#sec-meta-sourceProv...

you can slap a hash on a binary distribution and it becomes "reproducible" in the same trivial sense as any source tarball. after that, the reproducibility of whatever "build process" takes place to extract archives and shuffle assets around is no more or less fraught than any other package (probably less considering how much compilers have historically had to be brought to heel, especially before reproducibility was fashionable enough for it to enter much into compiler authors' consideration!!)

I'm curious, why couldn't packages that are fully reproduceable be marked with metadata, and in your config you set a flag to only allow reproduceable packages? Similar to the nonfree tag.

Then you'd have a 100% reproduceable OS if you have the flag set (assuming that required base packages are reproduceable)

You could definitely do that, I think the main thing stopping anyone is simply lack of demand for that specific feature. That, and also it might be hard to keep track of what things are properly reproducible; you can kind of only ever prove for sure that a package is not reproducible. It could be non-deterministic but only produce differences on different CPUs or an infinitesimally small percentage of times. Actually being able to assure determinism would be pretty amazing although I don't know how that could be achieved.

I assume it would be somewhat of a judgement call. I mean that is the case with nonfree packages as well - licenses and whatnot have to be evaluated. I assume that there are no cases of non-trivially large software packages in the wild that have been formally proven to be reproducible, but I could be wrong.

> It is closer to being fully reproducible than most other build systems (including Bazel).

How so? Bazel produces the same results for the same inputs.

Bazel doesn't guarantee bit-exact outputs, but also Bazel doesn't guarantee pure builds. It does have a sandbox that prevents some impurities, but for example it doesn't prevent things from going out to the network, or even accessing files from anywhere in the filesystem, if you use absolute paths. (Although, on Linux at least, Bazel does prevent you from modifying files outside of the sandbox directory.)

The Nix sandbox does completely obscure the host filesystem and limit network access to processes that can produce a bit-exact output only.

(Bazel also obviously uses the system compilers and headers. Nix does not.)

Uh, Either my understanding of Bazel is wrong, or everything you wrote is wrong.

Bazel absolutely prevents network access and filesystem access (reads) from builds. (only permitting explicit network includes from the WORKSPACE file, and access to files explicitly depended on in the BUILD files).

Maybe you can write some “rules_” for languages that violate this, but it is designed purposely to be hermetic and bit-perfect reproducible.

EDIT:

From the FAQ[0]:

> Will Bazel make my builds reproducible automatically?

> For Java and C++ binaries, yes, assuming you do not change the toolchain.

The issues with Docker's style of "reproducible" (meaning.. consistent environment; are also outlined in the same FAQ[1]

> Doesn’t Docker solve the reproducibility problems?

> Docker does not address reproducibility with regard to changes in the source code. Running Make with an imperfectly written Makefile inside a Docker container can still yield unpredictable results.

[0]: https://bazel.build/about/faq#will_bazel_make_my_builds_repr...

[1]: https://bazel.build/about/faq#doesn’t_docker_solve_the_repro...

I'm not familiar with Bazel at all so this might be obvious, but does Bazel check that the files listed in the BUILD file are the "right ones" (ex. through a checksum), and if so, is this always enforced (that is, this behavior cannot be disabled)?

The contents of files are basically hashed, if the contents don't change of the file listed for a target then no change will happen, even if you modify metadata of the file (like last modified time by `touch` or so on.)

Bazel is really sophisticated and I'd be lying if I said I understood it well, but I have spent time looking at it.

I think you're both right in a sense. Bazel doesn't (in general) prevent filesystem access, e.g. to library headers in /usr/include. If those headers change (maybe because a Debian package got upgraded or whatever), Bazel won't know it has to invalidate the build cache. I think the FAQ is still technically correct because upgrading the Debian package for a random library dependency counts as "chang[ing] the toolchain" in this context. But I don't think you'd call it hermetic by default.

Check out the previous discussion at https://news.ycombinator.com/item?id=23184843 and below:

> Under the hood there's a default auto-configured toolchain that finds whatever is installed locally in the system. Since it has no way of knowing what files an arbitrary "cc" might depend on, you lose hermeticity by using it.

AFAIK Bazel does not use the sandbox by default. Last time I experimented with it, the sandbox had some problematic holes, but I don’t remember exactly what, and it’s been a few years.

The very doc you link hints at that, while also giving many caveats where the build will become non-reproducible. So it boils down to “yes, but only if you configure it correctly and do things right”.

Yeah, I think you are right: by default, there is no OS-level sandboxing going on. According to documentation, the default spawn strategy is `local`[1], whereas it would need to be `sandboxed` for sandboxing to take effect.

Meanwhile, if you want to forcibly block network access for a specific action, you can pass `block-network` as an execution requirement[2]. You can also explicitly block network access with flags, using --nosandbox_default_allow_network[3]. Interestingly though, an action can also `require-network` to bypass this, and I don't think there's any way to account for that.

Maybe more importantly, Bazel lacks the concept of a fixed-output action, so when an impure action needs `require-network` the potentially-impure results could impact downstream dependents of actions.

I was still ultimately incorrect to say that Bazel's sandbox can't sandbox the network. The actual reality is that it can. If you do enable the sandbox, while it's not exactly pervasive through the entire ecosystem, it does look like a fair number of projects at least set the `block-network` tag--about 700 as of writing this[4]. I think the broader point I was making (that Nix adheres to a stronger standard of "hermetic" than Bazel) is ultimately true, but I did miss on a bit of nuance initially.

[1]: https://bazel.build/docs/user-manual#spawn-strategy

[2]: https://bazel.build/reference/be/common-definitions#common.t...

[3]: https://bazel.build/reference/command-line-reference#flag--s...

[4]: https://github.com/search?q=language%3Abzl+%22block-network%...

I believe your understanding of Bazel is wrong. I don't see any documentation that suggests the Bazel sandbox prevents the toolchain from accessing the network.

https://bazel.build/docs/sandboxing

(Actually, it can: that documentation suggests it's optionally supported, at least on the Linux sandbox. That said, it's optional. There's definitely actions that use the network on purpose and can't participate in this.)

This may seem pointless, because in many situations this would only matter in somewhat convoluted cases. In C++ the toolchain probably won't connect to the network. This isn't the case for e.g. Rust, where proc macros can access the network. (In practical terms, I believe the sqlx crate does this, connecting to a local Postgres instance to do type inference.) Likewise, you could do an absolute file inclusion, but that would be very much on purpose and not an accident. So it's reasonable to say that you get a level of reproducibility when you use Bazel for C++ builds...

Kind of. It's not bit-for-bit because it uses the system toolchain, which is just an arbitrary choice. On Darwin it's even more annoying: with XCode installed via Mac App Store, the XCode version can change transparently under Bazel in the background, entirely breaking the hermeticity, and require you to purge the Bazel cache (because the dependency graph will be wrong and break the build. Usually.)

Nix is different. The toolchain is built by Nix and undergoes the same sandboxed build process with sandboxing and cryptographically verified inputs. Bazel does not do that.

It does.

There are mechanisms for opting out/breaking that, just as with Nix or any other system.

> macOS

What does nix do on these systems?

Opt-out would be one thing, but it's actually opt-in for network isolation, and a project can disable all sandboxing with just a .bazelrc. Nix does have ways to opt-out of sandboxing, but you can't do it inside a Nix expression: if you ran Nix with sandbox = true, anything being able to escape or bypass the sandbox restrictions would be a security vulnerability and assigned a CVE. Disabling the sandbox can only be done by a trusted user, and it's entirely out-of-band from the builder. For Bazel, the sandbox is mostly just there to prevent accidental impurities, but it's not water tight by any means.

Ultimately, I still think that Nix provides a greater degree of isolation and reproducibility than Bazel overall, and especially out of the box, but I was definitely incorrect when I said that Bazel's sandbox doesn't/can't block the network. I did dive a little deeper into the nuances in another comment.[1]

> What does nix do on these systems?

On macOS, Nix is not exactly as solid as it is on Linux. It uses sandbox-exec for sandboxing, which achieves most of what the Nix sandbox does on Linux, except it disallows all networking rather than just isolated networking. (Some derivations need local network access, so you can opt-in to having local network access per-derivation. This still doesn't give Internet access, though: internet access still requires a fixed-output derivation.) There's definitely some room for improvement there but it will be hard to do too much better since xnu doesn't have anything similar to network namespaces afaik.

As for the toolchain, I'm not sure how the Nix bootstrap works on macOS. It seems like a lot of effort went in to making it work and it can function without XCode installed. (Can't find a source for this, but I was using it on a Mac Mini that I'm pretty sure didn't have XCode installed. So it clearly has its own hermetic toolchain setup just like Linux.)

[1]: https://news.ycombinator.com/item?id=43032285

> it's actually opt-in for network isolation

Bazel enables sandboxing by default, including network isolation. [1] [2]

The exception would be in environments that don't support it (Windows, unprivileged Docker container, etc.)

[1] https://bazel.build/docs/user-manual#spawn-strategy

[2] https://stackoverflow.com/a/63635141

My assertion that network isolation is opt-in is based on the fact that the --sandbox_default_allow_network defaults to true[1]. That suggests actions will have networking unless they are dispatched with `block-network`[2].

(It's hard to figure out exactly what's going on based on the documentation and some crawling around, but I wouldn't be surprised if specifically tests defaulted to blocking the network.)

[1]: https://bazel.build/reference/command-line-reference#flag--s...

[2]: https://bazel.build/reference/be/common-definitions#common.t...

I remember that a system nagged about non-reproducible outputs, Blaze (not Bazel, but the internal thing) allowed looking into the outside-world through bad Starlark rules and compile time tricks could get you questioning why there's so much evil in the world.

Maybe Bazel forbid these things right away and Googlers actually talking about Blaze will be inadvertently lying thinking they are similar enough.

I think talking about sandboxes is missing a point a bit.

It's an important constituent, but only complete OS-emulation with deterministic scheduling could (at a huge overhead) actually result in bit-by-bit reproducible artifacts with arbitrary build steps.

There are an endless source of impurities/randomness and most compilers haven't historically cared much about this.

The point I'm making is that neither Bazel nor Nix do that. However, sandboxing is still relevant, because if you still have impurities leaking from outside the closure of the build, you have bigger fish to fry than non-deterministic builds.

That all said, in practice, many of the cases where Nixpkgs builds are not deterministic are actually fairly trivial. Despite not being a specific goal necessarily, compilers are more deterministic than not, and in practice the sources of non-determinism are fewer than you'd think. Case in point, I'm pretty sure the vast majority of Nixpkgs packages that are bit-for-bit reproducible just kind of are by accident, because nothing in the build is actually non-deterministic. Many of the cases of non-deterministic builds are fairly trivial, such as things just linking in different orders depending on scheduling.

Running everything under a deterministic VM would probably be too slow and/or cumbersome, so I think Nix is the best it's going to get.

Sandboxing is relevant, but nix does that by default, so no difference here.

Nonetheless, I agree that Nix does the optimum here, full-on emulation would be prohibitively expensive.

You know, though, it would probably be possible to develop a pretty fast "deterministic" execution environment if you just limit execution to a single thread, still not resorting to full emulation. You'd still have to deal with differences between CPUs, but it would probably not be nearly as big of an issue. And it would slow down builds, but on the other hand, you can do a lot of them in parallel. This could be pretty interesting if you combined it with trying to integrate build system output directly into the Nix DAG, because then you could get back some of the intra-build parallelism, too. Wouldn't be applicable for Nixpkgs since it would require ugly IFD hacks, but might be interesting for a development setup.

Perhaps it's an area worth researching.

I don't know, concurrency is still on the table, especially that the OS also has timing events.

Say, a compiler uses multiple threads and even if you assign some fixed amount of fuel to each thread, and mandate that after n instructions, thread-2 must follow for another n, how would that work with, say, a kernel interrupt? Would that be emulated completely only at given fixed times?

But I do like the idea of running multiple builds in parallel to not take as big of a hit from single-threaded builds, though it would only increase throughput not latency.

> There are an endless source of impurities/randomness and most compilers haven't historically cared much about this.

The point is that Nix will catch a lot more of them than Bazel does, since Nix manages the toolchain used to build, whereas Bazel just runs the host system cc.

> It's an important constituent, but only complete OS-emulation with deterministic scheduling could (at a huge overhead)

This does actually exist; check out antithesis's product. I'm not sure how much is public information but their main service is a deterministic (...I'm not sure to what extent this is true, but that was the claim I heard) many-core vm on which otherwise difficult testing scenarios can be reproduced (clusters, databases, video games, maybe even kernels?) to observe bugs that only arise in extremely difficult to reproduce circumstances.

It does seem like overkill just to get a marginally more reproducible build system, though.

> Bazel also obviously uses the system compilers and headers. Nix does not.

Bazel allows hermetic toolchains, and uses it for most languages: Java, Python, Go, Rust, Node.js, etc. You can do the same for C++, but Bazel doesn't provide that out-of-the-box. [1]

Bazel sandboxing can restrict system access on Linux with --experimental_use_hermetic_linux_sandbox and --sandbox_add_mount_pair. [2]

Every "reproducible builds" discussion requires an understand of what is permitted to vary. E.g. Neither Nix nor Bazel attempts to make build products the same for x86 host environments vs ARM host environments. Bazel is less aggressive than Nix in that it does not (by default) attempt to make build products the same for different host C++ compilers.

[1] https://github.com/bazelbuild/bazel/discussions/18332

[2] https://bazel.build/reference/command-line-reference#flag--e...

No, most compilers are not themselves reproducible, even within very restrictive sandboxes (e.g. they may do some work concurrently and collect the results based on when it completes, then build on top of that. If they don't add a timing-insensitive sorting step, the resulting binary will (assuming no bugs) be functionally equivalent, but may not be bit-by-by equal), and a build tool can only do so much.

What are the common issues besides timestamps?

Compiler executing internal work concurrently and merging at the end. Thread scheduling changes will cause a different output ordering.

Reply on news.ycombinator.com