Hacker News Reader

Comment by Aardwolf | Hacker News Reader

Now just make "-j" the default and all is good :) Why shouldn't make use the available resources by default?

I really hate it when people do that and don’t provide a way to disable it.

There are a number of python libraries I can’t install on my laptop easily because the native code runs out of physical (4GB!) memory spawning compiler jobs and everything starts thrashing, it’s just dumb to go forking without asking the user first. People who really want it can give make the -j flag.

Bazel is the worst at this. When you try to eg; build tensorflow from source it maxes out all the cores, uses up as much memory as it can, and then crashes when the OOM killer eventually kills it. This is even on a 64GB 64 core machine.

Manually specifying resources doesn't work well either because it uses 2x+ the resources you allocate it.

You can do "taskset --cpu-list 0 make -j"

I've had a "make -j" fail due to virtual memory exhaustion (granted, on a 32-bit VM). This was on a C++ project, so it might be less of a problem when building the kernel, but it's definitely not a reasonable default across the board.

I've had similar issues, though it was actually caused by Ninja, and not Make. I was doing a build of clang+llvm itself, and somehow all the link steps ended up happening about the same time. It really exhausted the memory on my system. My system has 32 GiB, of which I rarely go above 10GiB used except for those giant builds.

Memory exhaustion for one. In my experience build systems will happily eat up all of your memory and put you into a swap hell that is difficult to escape from.

Especially painful when one of your coworkers has a beefier machine than you and changes the number of parallel jobs offhand one day, causing your computer to freeze until the OOM killer gets the right build process.

It exposes bugs in your make rules.

I prefer processes that expose bugs over those that obscure them...

I've learned over my engineering career to prefer rules which are based on the facts of the situation at hand rather than rules which are applied dogmatically. Dogma is what gave us Chernobyl.

You mean not using `-j` exposes bugs? Doubtful. If you are missing a dependency edge in your build graph then there are no guarantees that it will cause an issue with or without `-j`.

The only way to properly solve missing dependencies is to hide targets unless they are explicitly depended on. The only systems I know that do that are Nix and Bazel (and its clones).

If your rules have side-effects that aren't reflected in the dependency part of the declaration but nonetheless depended upon, running the jobs in parallel means the order of execution varies instead of being sequential, so sometimes you may get a file not found on something produced in an earlier rule.

If make knows about all the inputs and outputs of every rule, it can make sure dependencies are built before they are used. The problem occurs when the makefile doesn't tell make about every dependency.

Also the rule scripts themselves can have race conditions. A trivial example is if they end up using the same temp filename when doing work.

Yes I know. But the order of execution can vary even if you don't run in parallel.

For example if you have two targets, A and B, and B depends on A's output but you forgot to say so, then `make` can build B before A even if you don't use the `-j` option.

I thought the behavior I have seen is it's always in the order that the targets are specified.

So if you have:

all: A B

Then if both need a rebuild A is always first, except when -j breaks that.

But I can't say I have played close attention to every time I have typed make.

Ah right, but that's just a lucky coincidence. Make is perfectly within its rights to randomise the order of A and B even if you don't specify `-j`. And if you're driving Make from some other system, e.g. CMake, you really have no idea what order stuff is going to end up in the makefiles anyway.

Reply on news.ycombinator.com