Hacker News Reader

Not just a functional language -- I found that my ability to write maintainable (and performance) Makefiles drastically improved when I began to think of Make as a Prolog/Datalog variant with particularly heinous syntax and very limited constraint resolution.

I really like the declarative style of make, to the extent that I've been abusing it as an automation tool. Doing things like checking if a server is alive or setting up a serial port. But due to limitations like depending on mtime and issues with special characters in targets, I've switched some of my more intensive Makefiles to prolog using this: https://github.com/webstrand/robo.

When I find the time I hope to write bindings for netlink for prolog, so that I can declaratively create and configure network namespaces. Complete declarative system configuration using prolog would be a dream come true.

It's funny you mention that because I've wondered for a while what it would be like to have a Prolog-based build system (or even just as a CI config target instead of abusing yaml). I just lack the technical nous (and time) to try and pull it off.

I second Bazel. People keep on mentioning how steep the learning curve is, but the conceptual model is really simple, elegant, and intuitive.

What is steep is the technical know-hows:

1. When things don't work as expected. For example, while it worked flawlessly with languages that it natively support such as Java, that wasn't the case for other languages such as Javascript or Python.

2. When you have to do something advanced such as building custom Bazel rule. You'll need to understand Bazel internals and unfortunately the documentation isn't very intuitive and also somewhat sparse.

Bazel is awful unless you're Google and have a team of devs supporting Bazel.

I suspect the experience varies pretty widely based on what languages you use and what you’re trying to do with it. If you’re just building Go binaries then I’m sure Bazel is great, but if you’re doing anything with Python 3 or if you’re doing something a little off the beaten path like code generation, it’s probably a frustrating experience.

> If you’re just building Go binaries then I’m sure Bazel is great

Compared to other things in Bazel maybe, sure. But if you're just building Go binaries you don't need Bazel. The standard Go tooling is beyond sufficient: it's hard to beat.

Yeah, I completely agree.

Lots of things can be complicated in Bazel, but I would not at all consider code generation to be among them. You just write a genrule, and wrap it in a macro if you want to do the same thing more than once. (It can be more complicated than this, of course, but then code generation isn't the hard part of the problem.)

> I suspect the experience varies pretty widely based on what languages you use and what you’re trying to do with it.

I second GP's assessment that Bazel is pretty developer-hostile if you are not Google or have Google's resources to maintain Google's build systems.

Case in point: get Bazel to consume system libraries or vended/third-party dependencies in a C++ project. This is a pretty basic usecase, but it was left as an afterthought in Bazel, and it's an uphill battle just to get Bazel to work with them.

I don't doubt that Bazel can be usable if your entire world is locked tight in repos you control and were all forced to migrate to Bazel. It's the same thing with GYP, another developer-hostile build system spawned there. However, that is not how the world outside of Google works.

The only way I see Bazel become relevant and usable for non-Google/FANG-size orgs is if a higher level build system like CMake supports generating Bazel code, so that all these shortcomings are brushed under the proverbial rug. However, if we get to that, are we really using Bazel, or is Bazel relegated to an implementation detail?

Of course that is an overstatement, but:

I teach Bazel and help companies adopt it. I’ve noticed that a high percentage of those using Bazel have one or more xooglers on their team.

  > I teach Bazel and help companies adopt it.

The very fact that this occupation exists pushes me away from Bazel, having never used it. Makefiles are horrible, but they are well documented and I've been able to google my way out of every problem that I've either run into, inadvertently caused, or had to debug from somebody else's mess (it's always a mess).

I don't actually want a better build tool. I want a tool that I spend as little time as possible thinking about.

But that’s not really how it works. Build systems exist in a complex domain — whether we like it or not, that complexity can’t really be hidden away, it will leak one way or another.

The complexity need not "leak" - it could be an explicit feature of the tool. Make kinds sorta does this, as the learning curve is steep from the beginning but stays constant. Tools that are easier to use at first but then smack the user with "leaked complexity" set the user up for situations he is not experienced enough to handle.

That might be fine in an enterprise setting where the user likely has a mentor or support contract to turn to. But it's not what I want at home.

The problem with Bazel has nothing to do with learning curve. It's a pretty simple model as far as build systems go.

It's that it requires a lot of boilerplate and has a very rigid nested structure... which compounds boilerplate, unless you venture into custom plugins/build rules.

Some of the basic ideas are absolutely right, e.g. separating the resolution of dependencies from the build, and purity, but it's just soooo much boilerplate.

Also Bazel is very "corporate", it's not designed for projects that want to be good FOSS unix citizens.

Very telling: "common c++ use cases" https://docs.bazel.build/versions/4.2.2/cpp-use-cases.html includes pulling googletest source from the web and linking a random shared object blob. Zero mentions of pkg-config.

The http_archive has a sha256-attribute to deal with the "random". You could add multiple urls to point to your local mirror as well. It's a way to pin external dependencies and get reproducible builds without checking in a big blob into your git. pkg-config in my experience is less reproducible and tends to pollute and be affected by system environment, unless you have some build guru who can setup chroot in your project.

Oh I didn't mean "random" as in "could change". In fact "random" was about the unspecified .so blob, not the http archive.

> pkg-config in my experience is less reproducible

But this is what I mean, you only want full reproducibility in a "corporate" environment. In a FOSS desktop environment you often specifically want to use whatever the system has.

I think many of the ideas/techniques behind Bazel are excellent: hermiticity, distributed caching, dependency analysis, massively parallel execution, etc. There are so many things every other build system out there can learn and apply from Bazel that could make them 10x more effective.

But IMHO Bazel's fundamental flaw is that it tries to be a one size fits all solution that's supposed to work for all languages and ecosystems. That pretty much guarantees that it will have more need for boilerplate/configuration, be less performant, have more bugs, and at the end of the day just offer a worse user experience than a build system that applies all of Bazel's underlying ideas/techniques while being designed solely for a single language/ecosystem and more deeply integrated into it.

I think the eventual best case scenario for Bazel is a jack of all trades, master of none, but in its current state, for most of the languages/ecosystems it claims to support, it would be way too generous to even call it a jack of the trade in terms of overall UX.

I see GNU Make as sitting on the opposite end of a spectrum than Git. Git is fascinating as a sophisticated, featureful system emerging from a simple fundamental concept (Merkle tree of file hashes), whereas Make is a mish-mash of concepts cobbled together to expose a particular set of user features (describe and execute a graph of executions in a factorized way).

Git emerged decades after other version control systems were first invented, and found a neat abstraction that covered the space.

That's why I have such hope for the BSalC paper and what can emerge from it.

I think not adding an arbitrary new declarative language is good design feature of redo. I think redo has other issues but personally redo with bash as the underlying glue works a lot better than make (in terms of maintaining more complete DAGs). Now I'm not saying it wouldn't be nice to get a declarative language which works WITH redo, but it would be important to keep it separate.

One thing redo is missing for me is support for building things in a different directory. It's a nice-to-have feature but it's difficult to envision how such a redo would work. Really overlays actually solve this problem, although a fuse based overlayfs would also be nice (so you can use it without needing to escalate privileges).

Have you given bazel a serious look? It hits quite a few of the points you mention. There's a steep learning curve and you kind of have to go all in with it, but once you're there it's quite nice.

Bazel is alright (read: generally better than the competition) as long as you're not doing something uncommon with your toolchain or hitting one of its many shortcomings, otherwise it can quickly become a nightmare. It doesn't even make it easy to access the darn executable you just compiled (!) which always baffles me—if I compiled a binary I don't want it hidden away in some random folder, I want it right there so I can distribute it. And with Python, etc. you hit more obstacles. Make degrades more "gradually" in a sense: it's easy to do arbitrary things in a 1-line recipe, but then it becomes harder to generalize and make it modular. Overall, Bazel definitely wins as the scale of your project grows, and it has tons of useful features, but it's got some ways to go before I'd regard it as "nice".

For the sake of readers:

Bazel does not put outputs in random folders, it puts them in folders that are a deterministic mapping from the build target plus relative path to the thing being built in the source directory. It is well-suited if you have a huge amount of source code with things being output, but one among many speed bumps when getting started.

(also depending on what you're doing, `bazel run <foo>` will invoke the binary directly, and `bazel build <foo>` will, if there's an output artifact, usually print its location)

I meant random in the colloquial sense (same sense as which a SHA256 is "random"), not a random variable.

I skimmed the paper. I use Make, extensively, but in a way I generally don’t see in the wild (for C/++):

1. I convert all paths to abspaths;

2. All intermediate products go into a temp dir;

3. All recipes are functions;

4. The set of objects are defined by a “find . -name ‘…’” for C sources;

5. Every C file depends on every header file in the repo;

6. Use ccache; and,

7. Deterministic build.

My big work project is ~20mm loc, including ~65 FW images (embedded OSes). Incremental compilation is ~300ms. Clean rebuilds are 30-40s; scratch builds are 8m. (All of this on ~4 year old MBP.)

Obviously, there are some design tricks to get raw compilation speed up — naive C++ only compiles at ~3kloc/s; it’s possible to get that up to 100kloc/s.

> Every C file depends on every header file in the repo.

So change in single header file always becomes a full build? Do you use pimpl or other tricks to avoid frequent changes to h-files, otherwise class members usually end up in the headers for simplicity and to avoid incomplete type problems on non-pointers unfortunately. Was this C or C++?

What's the difference between a clean rebuild and a scratch build?

Yes: touching any header file causes a full rebuild. Remember that ccache is "under the hood" doing memoization. So, a clean rebuild could still hit ccache, whereas a scratch build starts from an emoty ccache.

As I have been saying for years: make is the worst build system, except for all the others.

"As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement."

Unless I am mistaken, the author of https://cr.yp.to/redo.html never implemented the idea. He has since used only the Bourne shell and standard UNIX utilties as a build system instead of make. For example, consider the "do" script in this project: https://nacl.cr.yp.to

The "do" script was written circa 2011. The idea for "redo" was published circa 2003. The idea for "make" dates back to 1976. GNU Make was written around 1989. For the avoidance of doubt, I am not suggesting any of these build system ideas or implementations are better or worse than the others. That is for the reader to decide. I am only pointing out which one is the most recent, or most modern, if we use the word "modern" according to its dictionary definition.

I’m quite happy with ruby’s rake for my needs. I mostly use it to define pipelines, often involving data downloads and transformations into datasets for ML. It works well when you work with files, preferably containing a single or few entities each. Have been meaning to wrap some tools to make it handle databases into an extension.

I don't like using tabs in Makefile syntax. Tabs are in many cases (especially on print) indistinguishable from spaces.

Also, any "make" program for C should automatically detect dependencies between files because this information is present in the code and it doesn't make sense to duplicate it.

GNUMake allows for custom, non-tab leader characters.

https://stackoverflow.com/questions/2131213/can-you-make-val...

make doesn't really understand C or C dependencies, but the compilers usually do. Here is a minimal example that will take care of dependencies by passing -MD -MP to gcc or clang. The dependencies (*.d files) get generated as part of the normal gcc/clang invocations. There is no separate dependency generation step.

  $ cat Makefile
  CFLAGS := -MD -MP   
  hello: hello.o x.o y.o
  -include *.d

  $ echo '#include "x.h"' > x.c
  $ echo '#include "y.h"' > y.c
  $ touch x.h y.h
  $ echo 'int main(void) {}' > hello.c

  $ make
  cc -MD -MP   -c -o hello.o hello.c
  cc -MD -MP   -c -o x.o x.c
  cc -MD -MP   -c -o y.o y.c
  cc   hello.o x.o y.o   -o hello

  $ touch x.h

  $ make
  cc -MD -MP   -c -o x.o x.c
  cc   hello.o x.o y.o   -o hello

Actually, the compilers do not and you are doing less than half of the job there. They don't emit information about the non-existent files that would change the build if they were to suddenly exist, such as an "x.h" in one of several standard search directories in that example.

* http://jdebp.uk./FGA/introduction-to-redo.html#CompilerDefic...

Also they don't add an empty rule for the source file (like they can do for headers), so if you rename from e.g. foo.c to foo.cpp, which still produces a foo.o, your incremental build fails because the dep file still has the dep on foo.c which doesn't exist anymore.

The original sin here is generating the dependencies from the source files in the same step that was used to generate the object files. The only way to ensure that Make observes structural changes in the dependency DAG is if the graph is rebuilt.

Good old "make depend". If only it didn't add a significant overhead...

You can get make to only re-generate its rules incrementally based on what was changed.

https://www.gnu.org/software/make/manual/html_node/Remaking-...

Looks like gcc does not plan to do this https://gcc.gnu.org/bugzilla/show_bug.cgi?id=28810

If a /usr/include/x.h were to appear, wouldn't that have lower priority than the x.h in the current directory and therefore not change the build?

> Also, any "make" program for C should automatically detect dependencies between files because this information is present in the code and it doesn't make sense to duplicate it.

This requires that either you implement or shell out to a C/C++ parser as part of the dependency resolution step (which has to happen prior to the actual compilation). This will slow down your compilation a lot unless you're able to parse C/C++ as fast as your make/bazel/whatever language.

The idea behind redo, in contrast, is that "dependency resolution" is not a separate step. The dependencies can be generated at the same time as the compilation, by the compilation in a perfect world; through some post-processing in a less than perfect one.

Unfortunately, current C and C++ compilers do not emit all of the information that they possess internally, only less than half of it (the positive information about files found, not the larger amount of negative information about non-existent files that were searched and would change the build if they were to exist).

* http://jdebp.uk./FGA/introduction-to-redo.html

I guess my point is that in the case of some target a that depends on some target b existing (I'm not sure what the best example is here, I know that bazel takes advantage of this to parallelize compilation, but I'm not intimately familiar with c++ compilation internals, so it might be clearer with a generated file example).

You have a.cc that depends on b.cc, but some tool generates b.cc, so it may not exist (but `b.do` does). If I invoke `a.do` to compile `a.cc`, and don't directly declare b.cc as a dependency in the .do file, the theoretical compiler will resolve that it depends on b.cc, and note that that doesn't exist, and then what?

In the bazel world, since you define the dependency graph explicitly before invoking any compiler, you can't run into this issue, but in situations where you need to get dep info from the compiler...what happens with stuff like this?

Why do you need to distinguish tabs from spaces?

Something a little better than make is makepp [1], although it can be more complex. We use it at $work and it works pretty well.

[1] http://makepp.sourceforge.net/

Gradle is a good make replacement as well. The kotlin DSL is much more readable and easier to follow than makefiles and has a lot of additional features.

Except when stuff goes wrong. Then gradle is a right PITA because debuggability is essentially zero. And periodically a new version of gradle will break something non-obvious deep in your dependency tree.

I wonder if the reactive trend will hit make-like tools.

> All the alternative build systems wisely choose to tackle only some of these features.

But why should all of these features be tackled by a single piece of software? Especially seeing how Make, in itself, has proven wildly insufficient already... what, 30 years ago, when GNU autotools came onto the scene?

More specifically, why do you find combinations such as meson + ninja, or CMake + ninja, or even CMake + make as superior to just-Make?

That was exactly my point: perhaps it's best not to bundle all those features in a single piece of software.

I haven't tried all the alternative combinations because, ultimately, I only have so much time and interest, and their learning curve is significant.