Reflecting on the Shake Build System
neilmitchell.blogspot.com
neilmitchell.blogspot.com
Small Project Build Systems - https://neilmitchell.blogspot.com/2021/09/small-project-buil...
Huge Project Build Systems - https://neilmitchell.blogspot.com/2021/09/huge-project-build...
That being said I think it's an absolutely amazing Haskell library for building complex systems with scheduling jobs and caching of intermediate results. Once you use it you get the sense that every large codebase has an ad hoc poorly specified build system nestled inside of it. I described the experience of using it briefly here:
Care to explain what that means or post a ref?
This has nice properties that allow you to be sure your build rules are correct, caching your outputs based on inputs, and because you can cache things you can now remotely execute things safer.
hermetic == sandbox which only contains declared deps
I assume bazel implements this via containers?
Does that include compilers and OS?
Also, IMO, this only provides a false sense of safety unless you archive absolutely everything that's inside the sandbox.
Yes, exactly. It also can include other deps like networking. A perfect hermetic build will (for most of the build process) not have access to the network.
> I assume bazel implements this via containers?
Sort of. Bazel's code predates "containers" by ~6 years. Bazel is the build system used at Google (sprinkle on a large amount of asterisks here).
> Also, IMO, this only provides a false sense of safety unless you archive absolutely everything that's inside the sandbox.
Yes, that is what a Bazel monorepo does. You can also include your language's runtime and standard library as part of the build process. See Dropbox's repo for more details on how they do this with Python: https://github.com/dropbox/dbx_build_tools
Some more details about what Google does are available here: https://opensource.google/docs/thirdparty/#
Essentially, everything is built from source (more asterisks here). Since everything is built from source using Bazel you can guarantee that at a certain view of the source repository you can build a reproducible copy of all software in the repo (more asterisks).
One other nice property is you can guarantee that all software in the repo is using a compatible version of each dep: https://opensource.google/docs/thirdparty/oneversion/
The downside is that if you want to upgrade a library that has a breaking change this requires you update a lot of code. The upside is that doing that isn't actually that hard with the tooling available and you can guarantee that you can link together any two libraries in the repo (more asterisks) and have the build succeed without any dep version conflicts. Since you can do this some langauges with runtimes (like JVM language) can see massive speed boosts because they only need to compile changed libraries and can run a `java -cp ... your.jar` to load up deps. You can check out rules_docker for some examples of how this can be used.
They are similar but I think of Nix as being closer to Dockerfiles than to Bazel. Nix describes how to compose and build things, of multiple build systems, in a unixy environment. It sandboxes things and runs commands in those things like `make install` and then pulls the files those commands generates out and uses them for other things. This is very close to how people use Dockerfiles but very different to how someone would go about using Bazel. Bazel's main selling points are that it allows you to describe how your software is built by using application level concepts. Bazel has `library`, `binary`, and `test`. Regardless of what the language we're talking about any two programmers will probably know what these high level concepts mean and, due to conventions in rules, any developer - regardless if they have experience in the language in question - can understand what is going on.
Bazel is also much finer grained than something like Nix. For example, the NixPkg for abseil-py is defined here: https://github.com/NixOS/nixpkgs/blob/nixos-21.05/pkgs/devel...
The BUILD files for abseil-py are found in the repo. Something you'll notice is that there isn't a single build target for all of abseil-py. Instead, each python module or "library" has it's own target: https://github.com/abseil/abseil-py/blob/master/absl/BUILD#L...
This allows for some cool stuff when deploying your software. Suppose you import just the flags module from this repo. That means that your built `py_binary` artifact will only contain just the modules that you're actually importing. You can also see the dependancy graph for all of your source code by talking to Bazel's query API. Find a bug in a library? You can now instantly query every single `binary` that those source files went into without false positives. That's just an example of something you may want to do but really you now have a way to query the dep graph for all languages and tools so the sky is the limit.
> instead of needing a monorepo with every piece of software you can just grab a reproducible build from nixpkgs
You also don't need a monorepo for Bazel to work. You can use `http_archive` or `git_repository` to load specific WORKSPACEs. For example: https://abseil.io/docs/cpp/quickstart#set-up-a-bazel-workspa...
Also bzlmod is being worked on (a package manager that automates this stuff). This also has provisions for running all of the `_test` targets on deps after they're compiled by your toolchain and on your hardware. This validates that all importing and compiling worked as expected (which most build systems can't accommodate).
In the meantime I'll stick to Bazel thnak you very much.