I'm not a huge fan of Nix's expression language, but the system works well.
Decomposing your dependency tree into small derivation pose no problem with Nix. Mainly due to the fact it solves entirely the ABI problem and the diamond dependency problem.
You can also simulate incremental build in Nix-only by using multiple derivations over a single source tree and combine them together.
Using a gigantic pile of complexity like Bazel just for that seems kind of an overkill to me. No offense, I am just challenging here :)
If you need to chase a bug into a more fundamental library, say libc, libpng, or your stack's favourite JSON library, iterating on that code with plain nix ("package-based non-incremental builds") will now take 20 hours per recompilation. With an incremental build system, it takes seconds, which is obviously a game changer.
I've had to do this multiple times in the past, and had to employ big hacks to get around this limitation.
My project also uses non-incremental nix, and an incremental build for our single-package top-level code. We just tolerate the waiting time, but it's not ideal. This works because our project is relatively small and updates to dependencies are not super frequent, so we can spend the time to build ad-hoc hacks to shortcut those builds each time. We find this better than any other currently available way, but it's not ideal.
I'm not convinced that Bazel is the final answer to this (it focused almost entirely on its way to do things, ignoring the package-based world that makes up the open-source eco system). But there's definitely the need for something that addresses both sides, cleanly, and it hasn't been found yet.
I see the point but the '20 hour' build seems to me over-exaggerated.
- Builds in Nix can be parallelized both on single node (multi-core, multi-socket) and multi-node without problems
- Parallel builds 'just works' in Nix with the Nix's sandbox system and the hash-based package identification mechanism. Nothing related to the bug mess that can be a distcc deployment or parallel builds in Bazels.
- Binary cache give a pretty efficient way to reuse any successful past build without problem
- Modifications in "base" packages generally do not happen everyday scenarios.
- Iterations and debugging without majors recompilations can be done with the nix shell directly. Full rebuild are required only for binary shipping.
> I've had to do this multiple times in the past, and had to employ big hacks to get around this limitation.
I had to recompile entire software stack ( > 1500 software ) for exotic archs (U.S supercomputers) in the past. My way was just to burst it on many nodes: compute time is cheap nowadays (and nodes are not missing on supercomputers :) ) .
My time however is not cheap and most incremental builds system are sensible to side-effect (including Bazel) and not as reproducible as they claim to be: Debugging that is in my experience time consuming, much more than a bit a build time.
I do see the interest of incremental builds inter-components, but in the current state of the tooling, it is for me not worth the pain. Specially not the pain of Bazel that has a (very) long list of quirks.
Without smaller compilation units than the default mkDerivation-per-component-style in Nix you end up having _extremely_ long build/test cycles on every change. You could alleviate this by deferring to whatever build system is called in mkDerivation by providing a short-circuiting shell.nix for every subcomponent (eg. a shell.nix to drop you into somewhere where you can `cargo build`). But IMO that's not nearly as nice of an experience as a `nix-build -A some-component` or `bazel build //some/component`, especially when changes span multiple components.
> You can also simulate incremental build in Nix-only by using multiple derivations over a single source tree and combine them together.
Yes, you can put in the effort to replicate Bazel-style small-compilation-unit language rules in nix. But then your evaluation time will skyrocket, with nix spending dozen of seconds re-evaluating these tiny derivations before even getting to the build - this already happens when evaluating complex derivations like NixOS configuration builds. This also kills your build/test cycle. Plus, you end up polluting your /nix/store with thousands of tiny out-of-date artifacts that then need to be garbage collected (which also becomes slow when the graph grows).
Bazel's build/evaluation model (workspace rules -> targets -> build graph -> action graph -> artifact cache) and the fact that it runs as a server makes it much easier to have snappy rebuilds out of the box without any special consideration for derivation granularity or crutches like shell.nix-per-component. I'm not saying Bazel is perfect, and it's quite difficult to get running in an organization if you don't know it well, but once it's set up correctly it's pretty much the best development workflow I've ever used.
Nix is very good for packaging software that you barely change, eg. for packaging software, as it makes it very easy to hermetically build software that comes with its own build system. But from my experience it's not a great experience for building software you develop, as build times rapidly rise to the point of frustration.
> [...] excepted if you inherited of a giant mono-repo mess.
I'm not sure how to interpret this - both Nix and Bazel shine in monorepo setups (and are IMO a prerequisite to even consider using a monorepo). Even nixpkgs is its own monorepo, and thanks to that is probably the best contributing-to-a-Linux-distribution experience that exists, by a long shot. Are you instead using Nix with flakes accross multiple repositories? Does that alleviate evaluation times?
And you describe exactly what we do :)
That's not that bad. The multi-components scenario can also be scripted through some specific nix-shell targets that you configure for a local source usage. It is not perfect but it does the job.
I do consider that if multi-components modifications happen on regular base, it is anyway a sign of a problem: they should probably not be separated components in the first place.
> But then your evaluation time will skyrocket, with nix spending dozen of seconds re-evaluating these tiny derivations before even getting to the build - this already happens when evaluating complex derivations like NixOS configuration builds
We do that by using overlays for our package set ( > 250 packages) over the bigger package set (the entire nixpkgs) which is provided through Nix channels. That allows to leverage the Nix internal cache and keep evaluation under few seconds. Also we do use the nix-daemon.
> Plus, you end up polluting your /nix/store with thousands of tiny out-of-date artifacts that then need to be garbage collected (which also becomes slow when the graph grows).
That is true, but space is cheap :)
> [...] I'm not saying Bazel is perfect, and it's quite difficult to get running in an organization if you don't know it well, but once it's set up correctly it's pretty much the best development workflow I've ever used.
I understand the point of view but I have the opposite experience. Bazel is one of the worst build system I had to experience. The fact he tries to act both as a package manager and as a build system (CMake, make, configure) makes it conflict with almost everything else in the Open Source world which is not made from day 1 for Bazel: This, for me is unacceptable.
I will also pass under silence the long list of quirks it has and the instability of the previous versions.
> Even nixpkgs is its own monorepo, and thanks to that is probably the best contributing-to-a-Linux-distribution experience that exists, by a long shot.
Nix requires only its configuration (Nix's expression) to be in monorepo. At the opposite, Bazel design is centered around the source code being in monorepo. That is a major difference.
One centralize versioning, the other one centralize everything.
Modern Bazel rule sets are more pragmatic than that. Python rules can pull packages using `pip install`, java rules can pull from Maven, rust rules from crates.io, etc.
Having a hell of a time getting a Nix-built CUDA image to work with EKS GPUs though.
I agree that nix is somewhat painful to deal with, but Bazel essentially has no package management story at all. It would be nice to see one of the tools subsume the other, but that doesn't feel like something either project is actively addressing (ie. package management in bazel, or incremental/non-bash-driven builds in nix).
The only time I use the nix CLI tools is when I'm fumbling around trying to make changes to one of our .nix files (we only have a few). Otherwise nix is hidden behind the scenes and I deal only with Bazel.
Also, I get the impression that a Nix artifact is generally an entire library/package, whereas a Bazel artifact is often just one or a few files which fits well with how I prefer to use the build system. But I'm no Nix expert and that may not always be true.
If Bazel had a better external dependency story (maybe someday [0]) then I'd probably not use Nix. But currently the pair seems to work well.
[0] https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3Zu...
This explains clearly why you'd want to combine both: