Redo: A recursive, general-purpose build system
redo.readthedocs.io
redo.readthedocs.io
Make is really a complicated combination of these components:
- a dependency definition language with in-place evaluation and half-assed wildcards
- A functional language with extremely limited datatypes (effectively just space-separated strings)
- a clunky macro system
- A "distributed" topological execution engine (did you know Make can coordinate job slots across recursive executions of Make? It's impressive! And shouldn't have to exist!) [1]
All the alternative build systems wisely choose to tackle only some of these features. Redo is interesting in that it's a minimal system that leans on existing Unix tools to fill the gaps, but I can't say I'm impressed with using Shell as the scripting language, though it is arguably more useful than Make's home-grown language that too few people know. Edit: actually, redo's doesn't enforce Shell, it can be any executable. That's more interesting, maybe (flexibility also introduces complexity!)
The most interesting development in this space is the paper "Build Systems A La Carte" [2] by Mokhov, Mitchell, & Peyton Jones, who did a great job of breaking down build systems to their fundamental concepts, and along the way found one particular point of the design space that wasn't fulfilled (Table 2, p79:16). Unfortunately, I fear it'll be some years before something production-ready emerges from that research, (and I'm sure I'll be grumpy about some of their implementation details anyhow! Build systems do not a happy developer make)
[1] https://www.gnu.org/software/make/manual/html_node/Job-Slots...
[2] https://www.microsoft.com/en-us/research/uploads/prod/2018/0...
Git emerged decades after other version control systems were first invented, and found a neat abstraction that covered the space.
That's why I have such hope for the BSalC paper and what can emerge from it.
Bazel does not put outputs in random folders, it puts them in folders that are a deterministic mapping from the build target plus relative path to the thing being built in the source directory. It is well-suited if you have a huge amount of source code with things being output, but one among many speed bumps when getting started.
What is steep is the technical know-hows:
1. When things don't work as expected. For example, while it worked flawlessly with languages that it natively support such as Java, that wasn't the case for other languages such as Javascript or Python.
2. When you have to do something advanced such as building custom Bazel rule. You'll need to understand Bazel internals and unfortunately the documentation isn't very intuitive and also somewhat sparse.
Compared to other things in Bazel maybe, sure. But if you're just building Go binaries you don't need Bazel. The standard Go tooling is beyond sufficient: it's hard to beat.
I second GP's assessment that Bazel is pretty developer-hostile if you are not Google or have Google's resources to maintain Google's build systems.
Case in point: get Bazel to consume system libraries or vended/third-party dependencies in a C++ project. This is a pretty basic usecase, but it was left as an afterthought in Bazel, and it's an uphill battle just to get Bazel to work with them.
I don't doubt that Bazel can be usable if your entire world is locked tight in repos you control and were all forced to migrate to Bazel. It's the same thing with GYP, another developer-hostile build system spawned there. However, that is not how the world outside of Google works.
The only way I see Bazel become relevant and usable for non-Google/FANG-size orgs is if a higher level build system like CMake supports generating Bazel code, so that all these shortcomings are brushed under the proverbial rug. However, if we get to that, are we really using Bazel, or is Bazel relegated to an implementation detail?
I teach Bazel and help companies adopt it. I’ve noticed that a high percentage of those using Bazel have one or more xooglers on their team.
> I teach Bazel and help companies adopt it.
I don't actually want a better build tool. I want a tool that I spend as little time as possible thinking about.
That might be fine in an enterprise setting where the user likely has a mentor or support contract to turn to. But it's not what I want at home.
It's that it requires a lot of boilerplate and has a very rigid nested structure... which compounds boilerplate, unless you venture into custom plugins/build rules.
Some of the basic ideas are absolutely right, e.g. separating the resolution of dependencies from the build, and purity, but it's just soooo much boilerplate.
Very telling: "common c++ use cases" https://docs.bazel.build/versions/4.2.2/cpp-use-cases.html includes pulling googletest source from the web and linking a random shared object blob. Zero mentions of pkg-config.
> pkg-config in my experience is less reproducible
But this is what I mean, you only want full reproducibility in a "corporate" environment. In a FOSS desktop environment you often specifically want to use whatever the system has.
But IMHO Bazel's fundamental flaw is that it tries to be a one size fits all solution that's supposed to work for all languages and ecosystems. That pretty much guarantees that it will have more need for boilerplate/configuration, be less performant, have more bugs, and at the end of the day just offer a worse user experience than a build system that applies all of Bazel's underlying ideas/techniques while being designed solely for a single language/ecosystem and more deeply integrated into it.
I think the eventual best case scenario for Bazel is a jack of all trades, master of none, but in its current state, for most of the languages/ecosystems it claims to support, it would be way too generous to even call it a jack of the trade in terms of overall UX.
But why should all of these features be tackled by a single piece of software? Especially seeing how Make, in itself, has proven wildly insufficient already... what, 30 years ago, when GNU autotools came onto the scene?
More specifically, why do you find combinations such as meson + ninja, or CMake + ninja, or even CMake + make as superior to just-Make?
I haven't tried all the alternative combinations because, ultimately, I only have so much time and interest, and their learning curve is significant.
When I find the time I hope to write bindings for netlink for prolog, so that I can declaratively create and configure network namespaces. Complete declarative system configuration using prolog would be a dream come true.
Also, any "make" program for C should automatically detect dependencies between files because this information is present in the code and it doesn't make sense to duplicate it.
https://stackoverflow.com/questions/2131213/can-you-make-val...
$ cat Makefile
CFLAGS := -MD -MP
hello: hello.o x.o y.o
-include *.d
$ echo '#include "x.h"' > x.c
$ echo '#include "y.h"' > y.c
$ touch x.h y.h
$ echo 'int main(void) {}' > hello.c
$ make
cc -MD -MP -c -o hello.o hello.c
cc -MD -MP -c -o x.o x.c
cc -MD -MP -c -o y.o y.c
cc hello.o x.o y.o -o hello
$ touch x.h
$ make
cc -MD -MP -c -o x.o x.c
cc hello.o x.o y.o -o hello* http://jdebp.uk./FGA/introduction-to-redo.html#CompilerDefic...
https://www.gnu.org/software/make/manual/html_node/Remaking-...
This requires that either you implement or shell out to a C/C++ parser as part of the dependency resolution step (which has to happen prior to the actual compilation). This will slow down your compilation a lot unless you're able to parse C/C++ as fast as your make/bazel/whatever language.
Unfortunately, current C and C++ compilers do not emit all of the information that they possess internally, only less than half of it (the positive information about files found, not the larger amount of negative information about non-existent files that were searched and would change the build if they were to exist).
You have a.cc that depends on b.cc, but some tool generates b.cc, so it may not exist (but `b.do` does). If I invoke `a.do` to compile `a.cc`, and don't directly declare b.cc as a dependency in the .do file, the theoretical compiler will resolve that it depends on b.cc, and note that that doesn't exist, and then what?
In the bazel world, since you define the dependency graph explicitly before invoking any compiler, you can't run into this issue, but in situations where you need to get dep info from the compiler...what happens with stuff like this?
1. I convert all paths to abspaths;
2. All intermediate products go into a temp dir;
3. All recipes are functions;
4. The set of objects are defined by a “find . -name ‘…’” for C sources;
5. Every C file depends on every header file in the repo;
6. Use ccache; and,
7. Deterministic build.
My big work project is ~20mm loc, including ~65 FW images (embedded OSes). Incremental compilation is ~300ms. Clean rebuilds are 30-40s; scratch builds are 8m. (All of this on ~4 year old MBP.)
Obviously, there are some design tricks to get raw compilation speed up — naive C++ only compiles at ~3kloc/s; it’s possible to get that up to 100kloc/s.
So change in single header file always becomes a full build? Do you use pimpl or other tricks to avoid frequent changes to h-files, otherwise class members usually end up in the headers for simplicity and to avoid incomplete type problems on non-pointers unfortunately. Was this C or C++?
What's the difference between a clean rebuild and a scratch build?
One thing redo is missing for me is support for building things in a different directory. It's a nice-to-have feature but it's difficult to envision how such a redo would work. Really overlays actually solve this problem, although a fuse based overlayfs would also be nice (so you can use it without needing to escalate privileges).
Unless I am mistaken, the author of https://cr.yp.to/redo.html never implemented the idea. He has since used only the Bourne shell and standard UNIX utilties as a build system instead of make. For example, consider the "do" script in this project: https://nacl.cr.yp.to
The "do" script was written circa 2011. The idea for "redo" was published circa 2003. The idea for "make" dates back to 1976. GNU Make was written around 1989. For the avoidance of doubt, I am not suggesting any of these build system ideas or implementations are better or worse than the others. That is for the reader to decide. I am only pointing out which one is the most recent, or most modern, if we use the word "modern" according to its dictionary definition.
Plus it’s worth considering that you can potentially use Fennel to configure the builds (since Fennel compiles to Lua).
Tup: https://github.com/gittup/tup
Fennel: https://fennel-lang.org/
Edit: Generator is a term typically used in CMake and Meson for this, in Make I'd use a pattern rule mostly.
Does this fail to address your problem?
The use-case at that time was to convert a LaTeX file containing a TIKZ image (therefore having a .tikz file extension) to an SVG which required compiling the LaTeX code to PDF, cropping the PDF and converting it to SVG. Since there were multiple occurrences of this across the project, I wanted to have that as a reusable "function" in the build system. I did not want to write a dedicated script for this because of portability issues (Shell vs. Batch).
```
inputs = { 'file1.c', 'file2.c' }
outputs = { '%B.o' }
commandA = 'gcc %f -c -o %o'
commandB = 'gcc %f -o %o'
objects = tup.foreach_rule(inputs, commandA, outputs) tup.rule(objects, commandB, {'app'})
```
Which apparently is a shortcut for saying:
```
tup.definerule{
inputs = {'file1.c'},
command = 'gcc file1.c -c -o file1.o',
outputs = {'file1.o'} tup.definerule{
inputs = {'file2.c'},
command = 'gcc file2.c -c -o file2.o',
outputs = {'file2.o'} tup.definerule{
inputs = {'file1.o', 'file2.o'},
command = 'gcc file1.o file2.o -c -o app',
outputs = {'app'} ```
Reference: http://web.archive.org/web/20201026140926/http://gittup.org/...
https://www.youtube.com/watch?v=zZ_nI9E9g0I&list=PLxj9UAX4Em...
Holding on to how things are done in the shell is not a thing to be proud of. I think a lot of us around here stopped counting the times we got tripped by globbing, forgetting or misplacing one special character in a ${} block, or quoting.
Let those monstrosities die already. Please.
There's this tool -- https://github.com/ejholmes/walk -- that is pretty good and I liked it but dropped it for the same reasons: it leaves the heavy lifting to you and it depends on your mastery in the black arts.
Now obviously I'm not managing huge projects but nowadays https://github.com/casey/just serves me just fine for everything I need.
I'm still not sure about having more than one file that's handling building however...
Will try it for sure though, thank you for the link.
For most projects the winning build system is not the most conceptually interesting one, nor the most flexible one. It's the most "convention over configuration" one. This is why Meson has taken the freedesktop/gnome/etc world by storm.
Another problem which is not discussed and is a very important feature for a modern build engine is "sandboxing" - forcing a build rule to declare all inputs, all outputs and to prevent any other kind of FS access and network access during the run of that command. BuildXL from Microsoft does that. Without sandboxing it's pretty hard to enforce a sane set of rules for building your large project.
[0] https://groups.google.com/g/tup-users/c/umW73zR5JKc?pli=1 . ex java creates numbered .class files for anonymous classes declared in a larger java file. In fairness, while looking, they may have relaxed some of that since I last looked with a transient flag https://github.com/gittup/tup/issues/405
Also, redo only lets a given .do file produce ONE output from a set of specified inputs. If you want to enforce this, nothing stops you from writing some declarative tool on top of redo (meaning you would replace the shebangs) which implements the sandboxing. Other than that, it's outside of the scope of concerns of a tool such as redo, and adding it would be a misfeature in my opinion (since it's not directly related to handling of the dependency graph).
1. How does redo compare with lower-level build tools such as ninja?
2. Why is it important/worthwhile to create a simplified/elegant version of Make, when for a few decades already, it is customary to write something higher-level that generates Makefiles (or builds in ways other than through Make, like scons or again via ninja)?
3. Is there a redo generator for CMake, and does it have advantages over Makefile generation?
1. Ninja is not meant to be hand written, redo can be hand written. Ninja verbosely describes your build as flat file containing the description of a DAG and the rules required to traverse it. Redo is a recursive definition, intended to be mostly hand written, which describes your build process. Redo can and should be split up across multiple files.
2. It may be customary to you, but not everyone agrees that the correct solution to building a project is to have a gigantic software suite consume a confusing and abstract configuration file and spit out a bunch of buggy makefiles. There are in fact plenty of people who hand write makefiles, redo fits in that same space. It's easier to hand write, it's easier to fully describe the DAG of your project, it's easier to reason about the build process and it's easier to make sure it is correct.
3. No. It would make no sense to have a redo generator for CMake as it would be severely missing the point of redo. The only reason CMake has generators for anything but ninja is because some people worry about portability to other systems and as such having make is better than nothing.
Most build systems seem to be like that, adding features on top of features. The end result is usually complicated and finicky.
I think that's why build systems are among the software developers hate most. And I do mean hate. (The first time and only time I saw an acquaintance, a normally calm person, have an outburst was talking about a build system.)
I think the ideal build system will be subtractive, not additive.
So I guess this is as good a time as any to talk about my current project, a build system. :)
(Most of this will be copied from two comments I made on lobste.rs.)
First, there is a paper that already lays out the features needed for an optimal build system: A Sound and Optimal Incremental Build System with Dynamic Dependencies. [2] The features are:
1. Dynamic dependencies.
2. Flexible "file stamps."
3. Targets and build scripts are Turing-compete.
That's it. Really.
The first item is needed based on the fact that dependencies can change based on the configuration needed for the build of a package. Say you have a package that needs libcurl, but only if users enable network features.
It is also needed to import targets from another build. I’ll use the libcurl example above. If your package’s build target has libcurl as a dependency, then it should be able to import libcurl’s build files and then continue the build making the dependencies of libcurl’s build target dependencies of your package’s build targets.
In other words, dynamic dependencies allow a build to properly import the builds of its dependencies.
The second item is the secret sauce and is, I believe, the greatest idea from the paper. The paper calls them “file stamps,” and I call them “stampers.” They are basically arbitrary code that returns a Boolean showing whether or not a target needs updating or not.
A Make-like target’s stampers would check if the file mtime is less than any of its dependencies. A more sophisticated one might check that any file attributes of a target’s dependencies have changed. Another might hash a file.
The third is needed because otherwise, you can’t express some builds, but tying it with dynamic dependencies is also the bridge between building in the large (package managers) and building in the small (“normal” build systems).
Why does this tie it all together? Well, first consider trying to implement a network-based caching system. In most build systems, it’s a special thing, but in a build system with the above the things, you just need write a target that:
1. Uses a custom stamper that checks the hash of a file, and if it is changed, checks the network for a cached built version of the new version of the file.
2. If such a cache version exists, make updating that target mean downloading the cache version; otherwise, make updating the target mean building it as normal.
Voila! Caching in the build system with no special code.
That, plus being able to import targets from other build files is what ties packages together and what allows the build system to tie package management and software building together.
I’ll leave it as an exercise to the reader to figure out how such a design could be used to implement a Nix-like package manager.
(By the way, the paper uses special code and a special algorithm for handling circular dependencies. I think this is a bad idea. I think this problem is neatly solved by being able to run arbitrary code. Just put mutually dependent targets into the same target, which means targets need to allow multiple outputs, and loop until they reach a fixed point.)
So how would this build system fit into the table (page 27, Table 2) of the "Build Systems a la Carte" paper?
It fits 8 of the 12 slots.
This is done with another feature: power-limiting. It will be possible to tell my build system to restrict itself, and any attempt to go beyond would result in an error. (This is useful in many situations, but especially to help new people in a team.) My build system can restrict itself along three axes: dependencies, stampers, and code.
To implement dynamic dependencies, you need a suspending scheduler, so obviously, my build system
The stampers are actually what defines the rebuilding strategy (and it can be different for each and every target), so there could be stampers for all of the rebuilding strategies.
This, technically, my build system could fill all four slots under the “Suspending” scheduler strategy in the far right column in table 2 on page 27.
In fact, packages will probably be build files that use deep constructive traces, thus making my build system act like Nix for packages, while in-project build files will use any of the other three strategies as appropriate. For example, a massive project run by Google would probably use “Constructive Traces” for caching and farming out to a build farm, medium projects would probably use “Verifying Traces” to ensure the flakiness of mtime didn’t cause unnecessary cleans, and small projects would use “Dirty Bit” because the build would be fast enough that flakiness wouldn’t matter.
This will be what makes my build system solve the problem of scaling from the smallest builds [3] to medium builds [4] to the biggest builds [5]. That is, if it actually does solve the scaling problem, which is a BIG “if”. I hope and think it will, but ideas are cheap; execution is everything.
Additionally, you can turn off dynamic dependencies, which would effectively make my build system use the “Topological” scheduler strategy. Combine that with the ability to fill all four rebuilder strategy slots, and my build system will be able to fill 8 out of the 12.
Filling the other four is not necessary because anything you can do with a “Restarting” scheduler you can do with a “Suspending” scheduler and Turing-complete code. (Just use a loop or a stamper recheck as necessary to simulate restarting.) And restarting can be more complicated to implement. So in essence, my build system will fill every category in the BSalC paper.
And I could do that because I took things away instead of adding more.
[1]: https://www.nature.com/articles/d41586-021-00592-0
[2]: https://www.informatik.uni-marburg.de/~seba/publications/plu...
[3]: https://neilmitchell.blogspot.com/2021/09/small-project-buil...
[4]: https://neilmitchell.blogspot.com/2021/09/reflecting-on-shak...
[5]: https://neilmitchell.blogspot.com/2021/09/huge-project-build...
1. Redo doesn't give you the tools to have dynamic dependencies for example, but it also doesn't concern itself with those and doesn't prevent you from having them. I have a small project which has dynamic dependencies (not system dependencies but just configuration options) and handles them gracefully with redo (without ever needing a make clean).
2. Flexible file stamps are something which you can implement on some redo implementation using a combination of redo-always and redo-stamp.
3. The targets and build scripts being turing complete is something which redo doesn't specify but that is the intentional use of redo.
That being said, I think there is a place for something even more BASIC than redo which can be used more directly to implement what you've described while also being flexible enough to build redo on top of. It has given me things to think about.
You're not wrong, and I think DJB did get there. However, there's a difference between the Unix way of designing a system out of disparate pieces working together (something DJB is good at and tends to do) and designing a cohesive whole system.
Obviously, redo is the former. My build system will be the latter, and I am doing it that way (despite being a fan of the Unix way in general) in an attempt to 1) hopefully make a build system that the masses actually like, 2) make it easier to make it work on Windows from the start, and 3) a cohesive whole build system is a much better platform for implementing hermetic builds.
(There are a bunch of smaller reasons, such as separating file targets and non-file targets, but all of those reasons are details.)
So yes, you are absolutely correct that redo can do everything I mentioned, and I guess that's an L for me. I should have mentioned other features as well.
> 1. Redo doesn't give you the tools to have dynamic dependencies for example, but it also doesn't concern itself with those and doesn't prevent you from having them. I have a small project which has dynamic dependencies (not system dependencies but just configuration options) and handles them gracefully with redo (without ever needing a make clean).
That's not quite what I envision when I say "dynamic dependencies" (my build system will also have configuration, but the config changing will not use dynamic dependencies), but yes.
However, at least in the TFA redo, dynamic dependencies (or simulating them) can be awkward. [1]
> 2. Flexible file stamps are something which you can implement on some redo implementation using a combination of redo-always and redo-stamp.
Correct. And I should have mentioned that. But again, DJB did that in an indirect way, making the system "finicky" (in my opinion).
> 3. The targets and build scripts being turing complete is something which redo doesn't specify but that is the intentional use of redo.
Yeah, no argument there.
> That being said, I think there is a place for something even more BASIC than redo which can be used more directly to implement what you've described while also being flexible enough to build redo on top of. It has given me things to think about.
Well, I don't think going even more basic is good here. Hermetic builds are just too good to pass up, and that requires some complexity.
[1]: https://redo.readthedocs.io/en/latest/FAQImpl/#can-a-do-file...
The problem with working on windows is that there's no shared standard (unless you count whatever irrelevant part of the POSIX spec that windows follows a shared standard). On windows there is no bash or posix shell, there is no gcc or a compiler which takes gcc-like options, there's no posix compliant or gnu compliant make. It's all just wrong. If you plan on making a build system, it has to take a lot of things and provide an abstraction layer over them to make them work for windows. I say forget it, it's a waste of effort, just make it work on linux and by extension (through WSL or mingw or whatever) you get windows. The alternative is worse as you have to add support for every special language to your build system. One of the best tests of a build system I've found has been to get a build system to build an executable and run it to produce an output which is then consumed as part of the build process. With pure make, this is purely awful and requires either generating a bunch of rules or lots of boilerplate, and generally it's easy to get wrong. With redo I got this working with relatively little work and got it to work with cross-builds.
> cohesive whole build system is a much better platform for implementing hermetic builds
In my adventures with redo, the biggest obstacle to hermetic builds that I've found aside from just the environment (env vars) has been compilers. Compilers suck at a: telling you what they're looking for and b: where they're looking for it. The best solution seems to be to strace gcc and look for calls to fstat or something like that to figure out which candidate header files it's looking for. And I hope you agree, that's an awful solution. Surely a better build system would need some form of overhaul to the interfaces provided by compilers? (Or, some re-implementation of some compiler features so you can spoon-feed a compiler the exact flags to ensure it behaves predictably even if the environment changes.)
I guess unifying things could solve the environment problem, but I think I can solve it without unifying anything.
>However, at least in the TFA redo
TFA?
>[1]
Okay, so you mean dynamically generated .do files. I think in most cases you can avoid them, but I agree that having every depend-er explicitly depend on the .do file before depending on the file it's intended to redo is really error prone and annoying. There's at least one way to retrofit this into normal redo: by having some default.o.redo file or something and wrapping redo-ifchange and checking if any dependencies have a default*.redo file and depending on the corresponding .do file, although kludgy this at least avoids the possibility of errors if you add future depend-ers to the dependency. I agree redo is missing something here though.
>Correct. And I should have mentioned that. But again, DJB did that in an indirect way, making the system "finicky" (in my opinion).
Why finnicky? In the usual case you DON'T want anything other than just timestamps and hashes, it works for all cases, albeit slowly for some. If you NEED to customize it, you have to pull out the special tools but then you just wrap that in some little utility and mostly forget about it.
> Well, I don't think going even more basic is good here. Hermetic builds are just too good to pass up, and that requires some complexity.
What do YOU think are the obstacles to hermetic builds? I already know what I've found but I'm always looking for more input.
I'm currently working with a few people to try to make redo work for strictly hermetic builds, or alternatively to design a better redo (or alternatively something else) which solves all the build problems I think exist. Unlike you, I don't really care if people like it or like using it, I've resigned myself to the fact that almost nobody cares about simple unix-esque software anymore and that if I want something like this, I'll have to design and write it myself.
Part of the language builtin to my build system provides ways of running commands without a shell.
> there is no gcc or a compiler which takes gcc-like options,
No, but there is MSVC. You just need to abstract the compiler and compiler flags. CMake did it, so why can't I?
> there's no posix compliant or gnu compliant make.
I don't need a make, and neither should redo. My build system will not be like CMake in that it will generate build files for something else; it will execute them itself. No make necessary. In fact, I could implement a make easily with my build system; just a parser would be necessary.
> If you plan on making a build system, it has to take a lot of things and provide an abstraction layer over them to make them work for windows. I say forget it, it's a waste of effort, just make it work on linux and by extension (through WSL or mingw or whatever) you get windows.
I don't think it's a waste of effort. People on Windows deserve a good build system too. In fact, not having a good one means that they are dependent on Microsoft. I'd like to help them reduce that dependence.
> One of the best tests of a build system I've found has been to get a build system to build an executable and run it to produce an output which is then consumed as part of the build process. With pure make, this is purely awful and requires either generating a bunch of rules or lots of boilerplate, and generally it's easy to get wrong. With redo I got this working with relatively little work and got it to work with cross-builds.
It's not hard to get make to do that. See [1] and [2].
> In my adventures with redo, the biggest obstacle to hermetic builds that I've found aside from just the environment (env vars) has been compilers.
Yes, but however it's done, it's still easiest in a self-contained system, IMO.
> Surely a better build system would need some form of overhaul to the interfaces provided by compilers? (Or, some re-implementation of some compiler features so you can spoon-feed a compiler the exact flags to ensure it behaves predictably even if the environment changes.)
The build system should carefully control the environment. In fact, it should make sure the environment is empty. (There are details, but in general...) That's the biggest thing to make sure the compiler does not act weird.
You should also do the build in a separate directory and copy needed files into the directory. Then check all commands and their arguments for possible files before running them. This is one reason why depending on an outside shell is not ideal.
Hermetic builds won't ever be perfect, but you can make it 80% of the way with only a few things.
> TFA?
"The Famous Article". It refers to the link under discussion on HN. In this case, it refers to the apenwarr redo.
> Okay, so you mean dynamically generated .do files.
Not just that. Those were just an example of how dynamic dependencies are already awkward in redo. There are countless other uses for dynamic dependencies. To give you a taste, my build system will actually be able to build a distributed system, i.e., it will be able to be used for DevOps deployment, and it will be easy.
Surely redo can as well, but as I've thought about how to do that, it seems awkward. Perhaps you can prove me wrong?
> Why finnicky? In the usual case you DON'T want anything other than just timestamps and hashes, it works for all cases, albeit slowly for some. If you NEED to customize it, you have to pull out the special tools but then you just wrap that in some little utility and mostly forget about it.
Finicky because as far as I can tell, the build database in redo is modified by multiple commands over a build. This can cause problems should a problem occur with the database mid-build. In a self-contained system, the database is in-memory during the entire build and only written out to disk once the build is done, which can then be used as a sort of "commit" of the build, marking it as finished.
There are other reasons, but that's the first that comes to mind.
> What do YOU think are the obstacles to hermetic builds? I already know what I've found but I'm always looking for more input.
I think a good paper to read would be the Dolstra thesis on Nix. [3] If you consider Nix a build system (not just a package manager), it goes further than even Bazel. The thesis should be a good place to start.
[1]: https://git.yzena.com/gavin/bc/src/branch/master/Makefile.in...
[2]: https://git.yzena.com/gavin/bc/src/branch/master/configure.s...
I didn't meant to imply that you or redo needs a make. Just that for something like autotools or cmake, there's a lot of abstraction required just to get things to work. Or you can bundle things or you can write your own things but the point is, it's a lot of work.
>I don't think it's a waste of effort. People on Windows deserve a good build system too. In fact, not having a good one means that they are dependent on Microsoft. I'd like to help them reduce that dependence.
Right, but as I point out, microsoft already took care of that by giving windows users WSL.
>It's not hard to get make to do that. See [1] and [2].
Yes I know what it takes, but if you don't think this is horrifically over-complicated then I don't know what to tell you. Instead of using implicit rules with target specific overrides you've made it slightly less boilerplaty by making it less scalable and expanding the entire build command for a specific target. I still think this is too much boilerplate and noise.
>Yes, but however it's done, it's still easiest in a self-contained system, IMO.
I think it may be easiest to implement (just like any complicated solution is easier to implement than a simple one) but I don't think it's a good goal for a build system. Complexity isn't free, even if it has a cheaper initial cost.
>The build system should carefully control the environment. In fact, it should make sure the environment is empty. (There are details, but in general...) That's the biggest thing to make sure the compiler does not act weird.
I disagree, you would also need to compile the compiler (because compilers like to take compile-time configuration). Moreover, you're going to have to have specific code for handling every different compiler with all its quirks, you're also going to have to spend a lot of time figuring out all the quirks. And you're going to have to spend a lot of time duplicating the code which implements those quirks.
Surely getting compilers to support some standard hermetic building facilities would be easier and better in the long run than maintaining a complex database of compiler quirks to handle for every specific compiler version etc?
> Hermetic builds won't ever be perfect, but you can make it 80% of the way with only a few things.
Yes but I can already get it 80% of the way with redo.
> Surely redo can as well, but as I've thought about how to do that, it seems awkward. Perhaps you can prove me wrong?
I'm still not sure we're both on the same page for what a "dynamic dependency" is but with redo I've solved the problem of having a dynamic .do file by simply depending on a dynamically generated .rc file: i.e a file containing sourcable shell variables (including bash arrays), generated using declare -p or things like the @Q parameter transformation.
>Finicky because as far as I can tell, the build database in redo is modified by multiple commands over a build. This can cause problems should a problem occur with the database mid-build. In a self-contained system, the database is in-memory during the entire build and only written out to disk once the build is done, which can then be used as a sort of "commit" of the build, marking it as finished.
There are no concurrency issues with the build database in any redo implementation worth it's salt and realistically there's nothing stopping a redo implementation from implementing its database the way you describe. It's also an implementation detail which I've never had to interact with. Not sure how that makes it finnicky.
> I think a good paper to read would be the Dolstra thesis on Nix. [3] If you consider Nix a build system (not just a package manager), it goes further than even Bazel. The thesis should be a good place to start.
From what I've seen, nix is too coarse grained for being a build system, which is where I've seen bezel recommended, but both of these fail to provide a general purpose build system which can be used outside a tightly controlled environment to build a piece of software hermetically using existing system dependencies. At the end of the day, I want to be able to build software with existing system dependencies provided by my package manager with the assurance that if I hit build, if those system dependencies have changed, the relevant parts of my code will be forced to rebuild and I will be told about it. I can see the value of totally hermetic builds, but a build system capable of solving the problem I describe should easily be extendable to solve the general problem of hermetic builds. In short, I want to be able to compile my code with my existing compiler, and if my compiler is updated, I want to be able to hit redo and have everything rebuilt correctly without worrying that something didn't get built.
I will say, though, that if you're not hearing about my build system in a few years, you can safely assume that you were right and that I was wrong.
It seems a bit defeatist to say this. Nobody on the internet, or the world for that matter, (unless they're agreeable and are just pretending to agree with you) is going to agree with you after the exchange of a few messages.
I should point out that I never disagreed with your fundamental claim that it would be easier to make a hermetic build system IF it's all encompassing and complicated. It's for example why systemd is easier to write if it's all tightly coupled versus if it was trying to have minimal coupling with minimal simple APIs and minimal simple tools building it up.
It's easier to write complex and tightly coupled software, it's why everyone does it instead of following the unix philosophy. I was simply pointing out that the unix philosophy was successful (for a while) for a reason, not because it was easy to follow it, but because the difficulty of making things simple paid off in the long run.
Please read what I wrote, I never made any hard claims about you being wrong, I've been mostly asking questions to get you to elaborate your stance. The sheer fact that I'm willing to entertain such long-form conversation should be enough proof of the fact that I haven't completely mentally dismissed your arguments as outright wrong.
> I will say, though, that if you're not hearing about my build system in a few years, you can safely assume that you were right and that I was wrong.
I don't think popularity (which is what you appear to be aiming for) is a good metric of the quality of software. Especially in the modern day and age where one of the most popular methods of shipping cross platform software is to ship it with its own instance of chromium and to write it as a local webapp. I think this is at least because modern day humans have become a lot more impatient and a lot more focused on short term convenience over long term benefits. But this discussion is outside of the scope.