Falsehoods programmers believe about build systems (2012)
pozorvlak.dreamwidth.org
pozorvlak.dreamwidth.org
You see, we all realized that just building code wasn't enough! What your 100 line JavaScript microservice really needs to be accompanied by is a build file that determines how to create an entire Linux VM. Also, because scale, your project also needs to know how to scale that VM out across multiple clusters!
The only answer of course is more build systems!
Oh but JavaScript sucks, so make that Typescript, so shove a transpiler in front of your scripting language. Because somehow we've gone down the path of needing build systems for interpreted languages.
I use all the above tech and get why it is there, but... seriously as a field this is the best we can do? :/
(I legit like Babel and TS, but you have to admit it is /weird/ that we're going through the complexity of having a build system but are reaping none of the performance benefits!)
And re: build systems for interpreted languages, what would the alternative be?
Asking Firefox and Chrome to integrate a language they didn't design?
Making it a runtime module like MoonScript, and letting everyone who loads the page pay for compilation, on battery-constrained computers, millions of times over?
Some build systems do build proper VM images, like AMIs.
These systems were built as incremental mitigations to old problems. I don't think it'll change until they collapse under their own weight and people switch over to something else completely.
A bunch of stuff I see comes with a dockerfile these days.
While I get building in docker for reproducibility, kind of -- shipping with one is a pretty big red flag. It means that it's likely to be very painful to port the code out of docker and integrate it into some other build. Which means that it'll be painful to integrate it with other code. Leave the build environment set up to the users. Embrace build diversity, it makes your code less fragile.
Those are the times I wonder whether I’m getting old. I kept using a shell alias that does ssh -ND, even though that requires maintaining a config file.
But on the other hand, is a dependency on having Docker available really worse for portability than having a dependency on having a reasonably standard Unix environment?
I often need to run it somewhere other than docker -- possibly integrating it into a different program.
Docker doesn't make this impossible, but it's a sign that portability was not considered when the code was written, and I'm in for pain.
No it wouldn't. Dockerfiles are stupid simple. A bunch of ADD/COPY commands to add files, and a bunch of RUN commands that are basically wrapping shell scripts.
Plus, find me a modern build system (CI/CD system, rather) that doesn't natively support Docker. It's become a lingua franca at this point.
I just wish it was faster on its own without me having to do all the legwork of maintaining lower layer containers etc. to avoid lengthy package install times.
A colleague predicted the problem and I learned about one more bit of random fragility that exists in today's ecosystem.
$ Docker system prune
It makes as much sense to complain about Docker because you let images accumulate in your system as it would make to criticize file systems because you never emptied your system's trash bin.
Why the hell should the very existance of unused docker images cause the single image I am using to peg multiple cores?
And no warning was given. Heck if my hard drive gets close enough to full that it's gonna be a problem my OS tells me.
Yes. And the code sitting beside them starts to make assumptions -- so when I try to port it to Android so that I can run it under the NDK, for example, it turns into a shitshow because I discover that the code depends on the quirks of the docker environment, and I need to untangle a whole bunch of issues to just get it to build on anywhere.
Containers -- dockers, jails, zones, etc -- are useful tools, but I'd rather see them left up to the people doing deployment, rather than the people shipping libraries. As a library author, I don't know where people want to run my code, so I serve them better by keeping the code portable.
Client side Babel makes sense. And packagers make sense. And it all unfortunately makes sense, but that doesn't make it good.
(Ok being able to try out new syntax with Babel is cool)
I think it can be done better, and above all, much simpler.
I think "redo" looks like a very simple, good solution - here is the documentation: https://redo.readthedocs.io/en/latest/
Of course it is only a tool and it does not convert an overcomplicated bloated system into a three-liner. But unnécessary complexity in build systems sucks big time, and interferes with decent testing, so I think this is a real gem.
Lol. Whatever, python community...
Anyway, I don't think redo really provides an answer to the insanity that is the js build system(s). I'm also a bit of a DJB fan, like python as a language, etc. However, this implementation in python... eh... I'm not sure about it. It's annoying to require someone to have some particular version of python to utilize a simple build system. He uses SQLite instead of the file system as a database paradigm favored by DJB.
DJB's software is all about technical correctness and simplicity of design. There's nothing wrong with this per se, but it wouldn't surprise me in the least to run this on some version of python that it isn't quite coded for and have it fail in some weird way without me realizing it.
The whole "I'm moving my python code to C tomorrow" thing is becoming a bit of a smell IMHO.
Shell script or a compiled language would be preferable in my opinion.
I think that Python as a draft/concept language, for getting the semantics right, and after that rewriting it into a standalone C program is sound, and that is the stated plan for the project.
I think what originally matters is not how it is implemented, but but how it is defined, what are the exact semantics, this is what was the Python development about. Because it is very simple and around 3400 lines of python, creating additional implementations should be not too much work, and my understanding is that a C implementation is the next planned step.
> He uses SQLite instead of the file system as a database paradigm favored by DJB.
Speed is one issue here, it also needs to support efficient multi-processing. Based on my own limited experience with something similar, my personal favorite for this kind of thing is LMDB (https://en.wikipedia.org/wiki/LMDB), which is well-tested, but I guess it would be very hard to make that into a portable version which runs, for example, on Windows, too.
> There's nothing wrong with this per se, but it wouldn't surprise me in the least to run this on some version of python that it isn't quite coded for and have it fail in some weird way without me realizing it.
From following the list (https://groups.google.com/forum/#!forum/redo-list) and blog, my impression is that the main author is a bit of a quality and reliability fanatic. So, for that kind of project, exactly the right person. He provided a test suite which can be used to test alternative implementations, as there are already a few around.
> Shell script or a compiled language would be preferable in my opinion.
There exists a basic (non-parallel) implementation in shell. shell is however going to be a bit too slow for a "production" implementation for large projects where speed matters. My favorite language for this might be guile (I think it matches Guix well), or perhaps Common Lisp. For maximum portability, I believe C is really the best thing.
I'd have nothing to say against an ultra-fast implementation for POSIX systems, written in Rust which calls into LMDB. But while this might be beautiful from a technical perspective, I don't think that there will be a noticeable speed difference in normal use.
Alas, it seems like a ton of extra effort to have BUILD files everywhere that repeat the imports of their sibling TS files. And on top of that, how am I going to ensure they stay up to date? What if someone forgets to manually add a dependency to the BUILD file after they've already done an `import {} from ...`? What if I don't want to vendor dependencies?
Unfortunately, it seems like Bazel + TS could be really cool, but the amount of extra effort won't pay off because there are foot-guns all over the place.
https://www.microsoft.com/en-us/research/uploads/prod/2018/0...
Here is a teaser, the 1st table:
Build system | Persistent build information | Scheduler | Dependencies | Minimal | Cutoff | Cloud
Make | File modification times | Topological | Static | Yes | No | No
Excel | Dirty cells, calc chain | Restarting | Dynamic | No | No | No
Shake | Previous dependency graph | Suspending | Dynamic | Yes | Yes | No
Bazel | Cloud cache, command history | Restarting | Dynamic(∗) | No | Yes | Yes
It is a truth universally acknowledged that a shitty scripting language in possession of a large user base will be in search of a new scripting language that automatically generates the shitty scripting language.
But then you have two problems.
Any deficiency in a complex, widely used and difficult to change development tool will be fixed by adding an extra layer over it.
If the deficiency is something like complexity or lack of reliability that can not be fixed by adding extra layers, read this comment again.
And since cmake's inception, all cmake alternatives are still pure crap.
The definition of "well" is relative.
…including CMake. :-)
Although, to be fair, I think CMake did at least achieve to prove that creating a turing complete "jack of all trades" meta build system is not a good idea in pretty much every case (although I'd argue that boost had done that well in advance).
It's too sad that with the myriad of legacy patterns around in the C/C++ build space (in production), there is pretty much 0 chance of simplifying the problem for a majority of cases instead.
However, to be fair cmake's main problem is it's appalingly poor documentation and user experience. It's very telling that since the release of cmake 3 the main source of info on how to do cmake the right way is not anythig churned out by cmake, whose docs are pretty much unintelligible, but random blog posts that manage to put together basic idiomatic code snippets.
This poor state of affairs is reflected in collaborative sources of info such as stack overflow, where well-meaning users reiterate poor or straight-out wrong ways of doing things mainly because they don't and can't know better.
"I thought I was dumb until I realized it's the book".
Top Amazon review of the top book on CMake. And I am still wondering whether it is the book, or the tool.
However, if you are in the privileged position of being able to address every use case by compiling and/or cross-compiling from a nix or Guix setup for a project, I'd certainly go with that, too.
The problem there is that this is likely never the case if you can't just containerize your application or prescribe which system it will run on -- which holds true for a lot of software.
While yes, this is a falsehood... in practice, you generally want to figure out a way to break cycles and remove them from the build system. In general, make the user responsible for breaking cycles in the build system.
The only major use case for cycles that I can think of is self-hosting compilers.
I would add some additional falsehoods:
- It is always the right thing to rebuild files when the inputs have changed.
- The host and target systems are the same (you are never cross-compiling).
- The host and target systems are different (you are always cross-compiling).
- You can reliably figure out the version number of whatever you are building.
- The build steps always produce the correct outputs if they do not fail.
- You always want to build third-party dependencies from source.
- You never want to build third-party dependencies from source.
- The input files do not change while the build is running.
- All of the inputs are files (and not, say, the current timestamp).
- The entire build should use the same build settings (optimization / debug), you don't want to do a partial-debug, partial-optimized build.
- You don't want to rebuild when e.g. files in /usr/include change.
- You always want to rebuild when e.g. files in /usr/include change.
- Installing additional software will not break the build.
- Running the same build twice will result in the same output.
- Build products do not contain absolute paths.
- You can move an executable from one path to another and it will still work.
- You can delete object files and everything will still work.
Normal C code, not that I can think of. But it is not difficult to give a real-world example:
A LaTeX document with a table of contents and an index has a cyclic dependency: The final document, including its pagination, depends on the table of contents, and the table of contents depends on the final document.
One of the uglier problems of LaTeX, which people of the likes of Leslie Lamport and Donald Knuth should have been able to solve, one might think.
There are LaTeX documents which never reach a fixed point (I've made one by accident exactly once), and need slight adjustment.
Actually I think you can do it iteratively by never decreasing the space occupied by an element between iterations. Of course this might yield ugly results but at least it halts with a solution.
IME any time you need performance & stability with unknown input, it’s gonna end up looking like streaming.
I have written LaTeX rules for Bazel.
- Use separate page numbering for the table of contents.
- Since the number of table of contents entries will not change, just use two passes; you don't need a separate page numbering for the table of contents.
- Never reference page numbers within the main text; use section numbers instead. This avoids problems with their size changed.
Anyways, when I need to do this, I do it purely in TeX rather than using a separate build system. (There is a way to use insertions containing alternating marks and penalties to extract the information needed without shipping out the page.)
Yup. In the build system for a self-hosting compiler I wrote I solved it using an optional target. If the compiler is present, generate the boot code using it, else download the boot code from a site on the internet, do md5sum checking and bootstrap from that. In a good build system, modeling this is a piece of cake.
- If build system X is used, people will read its documentation and write build rules according to the documentation.
No, people will just copy what's already there. And if it pulls in 35 external dependencies, compile 500 unnecessary files, and fails 25% of the time because of race conditions between external dependencies, people will add "Note: if build fails, run again."
Sigh.
You'd tend to think people learn over time but no. It's gotten worse. Right when you thought the industry was on the cusp of a build revolution the go community decides to push an old tool that most people can't utilize even 3% of when paired with `go build`. I kinda blame the go ecosystem for pushing Makefile adoption when they already have a caching build tool just so the maintainers didn't have to give the compiler a polished UI. Simplicity at the cost of functionality even if it's simply broken amirite?
Sigh.
- It's OK to embed/vendor dependencies
- It's OK to hardcode VCS URLs for your dependencies
- you can pull random stuff from the Internet at compile time and large companies will accept it
- It's OK to release libraries without version numbering or using git commitish
- Everybody uses containers
- Nobody cares about reproducible builds
- Nobody cares about security, license compliance and being able to rebuild things reliably
- All FAANGS will use your project even if it cannot packaged into OS packages
It now takes me weeks to setup a fresh machine to do local dev on. I guess I should make a docker image of my dev box.
cd devbox
docker-compose up
If a program doesn't handle my name, that's a problem with the program. My name is my name; the program doesn't decide that. Depending on the context some government might; the program also generally isn't supposed to decide for the government. Often in practice the program controls both of us; this is the core complaint of "falsehoods" articles, that a program artifact is overstepping its ethical/legal/otherwise extra-computational authority.
But there's no moral demand that e.g. a program exit with 0 status even when it's failed, or that build trees should be cyclic. These are tools by programmers and for programmers concerning programs and when we run into issues it's usually because we chose a poor modeling tool (e.g. a build system should not be ideal to calculate the fixed point of a data processing pipeline, independent of any cycles) or because of a flat-out mistake (there's no reason a compiler can't be given some output directive other than inventor oversight).
Other "falsehoods" are criticism of the software encoding the falsehood, but I read probably 90% of the complaints here as equal or greater failings on the tools invoked. Not only for the sake of my build system do I want DAG pipelines, updated timestamps, or meaningful exit codes - if anything these are the demands I wish to place on other software for the sake of my brain! More like "falsehoods compilers believe about their primacy in program generation."
Or as one of the comments suggests,
> build systems are a symptom of software languages that are not designed to build software systems
falsehood #1
The worst are crackpot systems like ant. Whoever thought a half-baked DSL with a XML concrete syntax was a good idea?
> Whoever thought a half-baked DSL with a XML concrete syntax was a good idea?
You kinda answered your own question there no?
Make does so many things correctly:
- it gives user total freedom to modify the built
- yet it has excellent defaults for most situations
- it is full blown programming language,
- yet it's syntax is extremely specialized for the purpose and familiar at the same time (it's bash)
- it's basic assumptions/structure are extremely simple (timestamps, dependencies)
- yet it's extremely powerful and can take into account most situations.
It's be extremely Unix-style tool with very simple building blocks that combine together in extremely powerful way. Its fast to learn and easy to master and as such, the best kind of design.
Sometimes making a build with it is a small programming project, yet after using all kinds of built tools, I end up just wishing that I could use make instead.
> and familiar at the same time (it's bash)
Make is an orchestrator that defaults to sh (not bash, actually). But what I've seen in some really good ones is taking advantage of the "orchestrator" part rather than the "sh/bash" part: One-line recipes that simply call a short script in any other language, including python, that do the desired thing for that recipe. The Makefile is then just used for coordinating partial-running those scripts.
On top of that, you could in fact set SHELL/ONESHELL and then the recipes themselves are python inside the Makefile:
SHELL := python
.ONESHELL:
testfile:
import csv
writer = csv.writer(open("$@", 'w'))
writer.writerow(['one', 'two'])
writer.writerow(['three', 'four'])
$ make testfile
import csv
writer = csv.writer(open("testfile", 'w'))
writer.writerow(['one', 'two'])
writer.writerow(['three', 'four'])
$ cat testfile
one,two
three,four"Build tools are only for building software" is probably another falsehood to add, make can be a great tool anywhere you've got a dependency graph.
It also would need to basically support virtualenvs and pip install to be really compelling, I think. Python without those is a pale imitation. That CSV module is known to be pretty bad, for instance (like many built in modules).
For build it mostly works fine. Doing anything else with it would be a total pain.
Is this not true?!? While working on my build times I benchmarked that something like a quarter of my time in clang was due to my having large numbers of sparse -I folders as the stat calls were so expensive (this was a 12" MacBook running macOS 10.14 on, I assume, APFS).
However, if I'm dealing with something special, or if the build system is unstable, I always fall back to Make. Such a beautiful tool!
Tup is also worth to mention: http://gittup.org/tup/
For example, "Build graphs are acyclic." Bazel enforces this. "Every build step updates at least one file." If your build step doesn't produce output, Bazel will complain.
"It's possible to tell the compiler which file to write its output to." "It's possible to tell the compiler which directory to write its output to." Bazel does this with sandboxes.
1. build graphs are trees/acyclic : bazel enforces this, no cyclic dependencies allowed
2. Compilers will always modify the timestamps on every file they are expected to output : if a write happens why not update mtime? nanosecond precision, what year is it?
3. It's possible to tell the compiler which file to write its output to : makes no sense to me, we expect our compilers to be writing to the correct files in the correct directories.
4. It's possible to predict in advance which files the compiler will update : yes it is, this is how caching works again see (1)
5. It's possible to determine the dependencies of a target without building it : yes yes it's completely possible given (1)
6. Detecting changes via file hashes is never the right thing : Bazel i think uses a combination of hashes and stamps, but why would using hashes not work 99.99999999999999998% of the time?
7. Non-experts can reliably write portable shell script : lol, experts can't reliably write portable shell scripts either
8. Your build tool is a great opportunity to invent a whole new language: or just coopt pieces of one (python: starlark, skylark)
9. Said language does not need to be a full-featured programming language : yes, it doesn't (again .bzl files)
10. Said language should be based on textual expansion : not sure what the alternative to textual expansion (parsing?) is
11. Adding an Nth layer of textual expansion will fix the problems of the preceding N-1 layers : again how is this different from parsingIt has nanosecond precision, but on most systems it does not have nanosecond resolution.
$ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify:
Modify: 2020-07-18 21:37:56.421487075 +0000
Modify: 2020-07-18 21:37:56.421487075 +0000
$ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify:
Modify: 2020-07-18 21:37:57.397502229 +0000
Modify: 2020-07-18 21:37:57.397502229 +0000
$ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify:
Modify: 2020-07-18 21:37:58.273515840 +0000
Modify: 2020-07-18 21:37:58.277515902 +0000
$ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify:
Modify: 2020-07-18 21:41:00.736348218 +0000
Modify: 2020-07-18 21:41:00.736348218 +0000
$ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify:
Modify: 2020-07-18 21:41:01.708363288 +0000
Modify: 2020-07-18 21:41:01.708363288 +0000
$ touch a1; sleep .0001; touch a2; stat a1 a2 | grep Modify:
Modify: 2020-07-18 21:41:02.504375653 +0000
Modify: 2020-07-18 21:41:02.508375715 +0000
Here's a quick and dirty script to play with this:
C=0
R=0
while :
do
R=$((R+1))
touch a1; sleep $1; touch a2
T1=$(stat a1 | grep Modify | sed -e 's/.*://' | sed -e 's/ .*//')
T2=$(stat a2 | grep Modify | sed -e 's/.*://' | sed -e 's/ .*//')
if [ $T1 != $T2 ]; then
dc <<HERE
20k $T2 $T1 - p
HERE
C=$((C+1))
if [ $C -eq 100 ]; then
echo "$R runs"
exit 0
fi
fi
done
$ bash t.sh 0.001 | sort | uniq -c
4 .004000061
83 .004000062
13 .004000063
1 222 runs
$ bash t.sh 0.001 | sort | uniq -c
29 .003999599
44 .003999600
27 .003999601
1 323 runsJust realized they other day that PowerShell does this (parsing instead of expansion). Which means much less worrying about escaping output of arguments.
> Build graphs are trees. Build graphs are acyclic.
Bazel uses some tricks/magic to hide outputs of build rules from other build rules. Bazel also prevents cyclic dependencies (A depends on B which depends on A).
> Compilers will always modify the timestamps on every file they are expected to output.
Source files a target depends on will be hashed. The operations applied to the files will also be hashed.
> It's possible to determine the dependencies of a target without building it.
Bazel forces people to explicitly define all deps that the source code uses. It uses sandboxing to hide unspecified deps from the source code. All of the definition of these rules is defined statically-ish (if you ignore WORKSPACE which technically applies to Blaze).
> Targets do not depend on the rules used to build them.
Bazel hashes the operations the rules will perform.
> Targets depend on every rule in the whole build system.
Bazel understands the full DAG and knows what each target depends on and what depends on each target.
> Detecting changes via file hashes is always the right thing. Detecting changes via file hashes is never the right thing.
It's a tradeoff that someone has to decide. Neither is right or wrong. Using a file content+metadata hash allows you gauntness that the build is always correct.
> It's possible to narrow down the set of possibly-updated files to a small hand-enumerated set.
As long as you use file hashing and are explicit about your inputs and outputs of every build rule, you can.
> Nobody will ever want to rebuild a subset of the available dirty targets.
You can build a specific target and it's deps at any time. Unrelated dependency trees are ignored.
> stat is slow on modern filesystems.
Anything is slow given a large enough scale. Any syscall is slow once you have millions of source files.
> Non-experts can reliably write portable shell script.
In bazel people interact with higher level rules. You don't need to write shell scripts unless you're implementing something that requires shell scripts. If you'd like you can also use an output of a target as a tool to do further compiling. For example: build protoc then use protoc to compile protos into libraries.
Every build step updates at most one file. Every build step updates at least one file. It's possible to tell the compiler which file to write its output to. It's possible to tell the compiler which directory to write its output to. It's possible to predict in advance which files the compiler will update.
> Your build tool is a great opportunity to invent a whole new language. Said language does not need to be a full-featured programming language. In particular, said language does not need a module system more sophisticated than #include. Said language should be based on textual expansion. Adding an Nth layer of textual expansion will fix the problems of the preceding N-1 layers.
It is a great opportunity to implement a domain specific language as "building software" is a domain that we would like to define programs for. Bazel uses skylark, it's a "fully featured" programming language in that there's loops, conditionals, functions (called "macros"), and a module-ish system ("load()"). It does not operate on textual expansion.
> It's totally OK to spend over four hours calculating how much of a 25-minute build you should do.
Bazel's target analysis takes a while but for my scale of code base it doesn't take hours, it takes seconds. If you have a huge codebase then these "wasted" seconds would be earned back by it's effective caching and remote build execution.
> All the code you will ever need to compile is written in precisely one language.
Bazel is multilingual.
> Everything lives in a single repository.
What is a "repository"? It's really a file system. What is a file system? A fancy kv store. Could you build a KV store that could return the values for every key that is a file in your source tree? Probably.
Bazel is built around a monorepo design but you could just as easily clone libraries into shared folders and executre bazel from above your project. You could also use remote repository rules. IMO you're better off just vendoring any source code you're writing in a folder in your repo so you can guarantee old versions of things are around when you need them and you never have a leftpad incident.
> Files only ever get updated with timestamps by a single machine. Version control systems will always update the timestamp on a file. Version control systems will never update the timestamp on a file. Version control systems will never change the time to one earlier than the previous timestamp.
File contents are check so this is not as much of an issue.
> Programmers don't want a system for writing build scripts; they want a system for writing systems that write build scripts.
I think people don't want systems like this because they've been burned. I've used a lot of makefile generators and they're all a really poor caching system. You essentially make cache invalidation part of your dev flow. AS a developer I don't want to think about that. I want to ask something to build me something and have it know when it needs to reload build definitions or code. I also don't want to care about what specific "gcc" or "javac" commands are being run. I need to be able to go down to that level but 99% of the time I just need to say "Turn a.c, b.c, and c.c into foobar.bin".
As for what bazel is really bad at, these all still apply:
> People will only want to build software on Linux. People will only want to build software on a Unix derivative. Nobody will want to build software on Windows. People will only want to build software on Windows.
Bazel support in windows is shaky. Bazel itself works but many tools (rules_docker) do not work. One place bazel would be really amazing for, in my opinion, is the game development industry. You have massive source trees, very complex packaging steps, you often have tools that are used as a compiler further in the compilation process (asset packets), you can benefit from shared build clusters, etc.
> Single-character magic variables are a good idea in a language that most programmers will rarely use.
Not the same but make-like variable substitution exists and it's a bit of a pain: https://docs.bazel.build/versions/master/be/make-variables.h...
> System libraries and globally-installed tools never change. Version numbers of system libraries and globally-installed tools only ever increase.
Bazel is mostly hermetic except for extremely strange things. Build toolchains that are "expected" to be in the host bleed in and effect a lot of things. It's a really gross side effect of certain things and can cause some nasty bugs.
And for some new rules that bazel sins with:
1. People are ok with running builds on hosts of separate architectures to build their software for different architectures
Cross compiling in bazel is extremely painful.
2. IDE integration doesn't matter.
A lot of people who use bazel don't really care about how well, or how poorly, it integrates with an IDE. As such a lot of the open source world around bazel's IDE integrations are lack luster. Getting things like gRPC working in bazel with auto-completions for multiple languages is nearly impossible.
3. Everyone wants 100% reproducible builds.
Many people do not want reproducible builds. That may sound shocking to some but a lot of other tools depend on non-reproducible elements of a build chain. For instance: when using bazel to build a docker container it sets the container's creation timestamp to the unix epoch. Because of this I cannot use Gitlab's registry cleanup script as it'll wipe all images since they're all over N days old.
4. All of the compilers I want to use during my build are installed on my host system.
Docker was probably the most amazing improvement in build/tooling workflow in recent years. If I want to build something with docker all I need is docker. If bazel to make this workflow the same it could take over the world. If there was a single place to configure something like this:
toolchains(
arm: {
c: "gcc:10.1.0"
}
x86: {
c: "icc:....."
}
)
I don't know if this is the best way to represent this concept but I have no idea why I have to even be concerned what version my host OS ships of gcc. I also don't know why I have to care if my host system is x86 but I want to build a binary for arm. Either use cross compiling features of the compiler or spin up a VM and run the build using native tooling in there. I shouldn't have to manually do that.
If you're using a tool, you really should know how it works and where it will break. You should also choose the tool based on the problem you're trying to solve. If you've got a build system that rebuilds when /usr/include changes, and you're trying to do cross-compiles, choose another build system.
Sure ok maybe you touched a file and the timestamp wasn't updated (it should, and your tool is making things harder on you). Or if you're going by hash, same hash, same file. Period. (Unless you're doing something funny with file metadata, and then again, you're shooting yourself in the foot)
Same with some comments on the original article saying "tool doesn't return correct status codes for success/failure" well, maybe you need tools that work then
Most of the advice in the article is sound though
Why? What's wrong with Make?
I feel that when I hear somebody say they don't like make, usually they misunderstand this, or they think somebody's bad experiences with makefiles is somehow representative of the capability of the tool.
One person created make in a weekend in 1976. We are still talking about it. Most humans and most software will never achieve this.
Yes, I realize GNU Make has a config option to fix this. That doesn't seem to be widely adopted.
On a second thought, the shellisms it borrows are also annoying. POSIX shell syntax needs to die on its own right.
I reserve the right to be fully satisfied only after the majority of random Makefiles I find on the net or emitted from build systems will use that convention.
To add another thing to the pot, let me repeat some things already mentioned or implied:
Make is too focused on files. Not every input is a file, and not every artifact is supposed to be a file either.
Make doesn't allow choosing the update criterion per file. Sometimes it's the has, sometimes the timestamp, and sometimes the parsed AST of the source code that should trigger a rebuild when changed.
It's a good thing that the update criterion can be delegated, but this should really be a thing that has a wide range of builtin options, both for ergonomics, and for avoidng mistakes. "Goals" comes to mind: http://git.annexia.org/?p=libguestfs-talks.git;a=blob;f=2020...
The list is long, but here's a random sample:
- If you have an atomic step that produces multiple targets, it confuses make, and encoding the fact properly in the Makefile is *really* not straightforward.
- tabs are indeed a pain
- make's DSL (the part that pre-processes the Makefile before feeding it to whatever shell you want) is really ugly. In particular, using '$' in the commands is a disaster coz make will chew on those before passing them to the shell.
- Oh yeah, and $ABC is in fact understood as "evaluate variable A" and concatenate with the string "BC". Ugh.
- managing "generated" dependencies (.e.g. includes) is a pain, especially if you try to have make manage the generating step
- managing dependencies on the actual content of the Makefile itself is a pain (as in: if I change the Makefile, what should be rebuilt?)
- Make has no (or rather very little) introspection capabilities and even less self-introspection capabilities. If you want a proof of that fact, try to build a graphviz graph of the dependencies, or better: know what commands will be executed if you change a specific file.
- The many warts an bags on the side that make grew over the years (.PHONY anyone?)
- Recursive Makefiles are a nightmare and make is simply not equipped to deal with them
- etc...Make does have a lot of warts, but this isn’t one of them any more. Grouped targets were added in the most recent release of GNU Make, and they address exactly this problem.
Re: your other points, many of them boil down to “the syntax is weird”. I agree 100%, and also it’s not so tough once you get used to it. I use GNU Make for tons of things, and I like it a lot.
Very good to know, thank you (that's indeed in release 4.3 IIUC)
> e: your other points, many of them boil down to “the syntax is weird”.
I don't think lack of introspection or recursive Makfiles or generated deps. falls in the "syntax is weird" column. Weird syntax I can leave with. Lack or very difficult to use essential features are a different story.
How did this go wrong??
Then there were some portability issues that should have been handled by the compiler, but by that time there were already metabuilders, so nobody bothered.
Simplifying assumptions are good.
I mean what do you expect? People cram Turing completeness everywhere and anywhere.
As in any mutually assured destruction scenario, we must all restrain ourselves.
https://news.ycombinator.com/item?id=7609289
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...