An optimization to help code compilation times on big CPUs | Hacker News Reader

An optimization to help code compilation times on big CPUs

phoronix.com

145 pointseaguyhn42 comments

the git commit: pipe: use exclusive waits when reading or writing[1]

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

I am always amazed at the quality of linux kernel commit messages.

Indeed, my personal rule of thumb is if the ratio of commit message to diff is greater than one then it’s a good commit. Larger the ratio better the commit and you would do better to read the commit (as opposed to skimming through them).

Chances are, you’ll learn something new.

Indeed. Those two lines in the test-case are particularly excellent:

  /* 64 processes */
  fork(); fork(); fork(); fork(); fork(); fork();

I can totally see most people, myself included, writing a dozen of lines instead, with a for loop, checking for parent/child/error return code, etc. Which might be preferable for some code somewhere deep inside the app logic that will potentially run on a resource-constrained machine. But for the explanation code to be read by a human, this one-liner is such a good choice!

From the linked text:

> In particular, the GNU make jobserver code ends up using a pipe as a way to limit parallelism, where each job consumes a token by reading a byte from the jobserver pipe, and releases the token by writing a byte back to the pipe.

Umm, that's what semaphores are for.

GNU make runs on a lot of different systems. There is not a semaphore API shared between them, but there will (almost certainly) be a usable pipe() on all of them.

But… if I could, I would probably use POSIX semaphores, no wait SYS-V, no, hang on, let me read some web pages… um… let me check my repositories and see which I've used before… both… hmm…

Seriously, I can't see getting so frustrated choosing that I'd just go with pipe(), but there is room for improvement in portable semaphore APIs.

I tried quickly to find the list of supported systems, but the main documentation pages don't mention them.

That said, pipe() is not "portable" either, so whether to use platform-dependent code for pipes or semaphores... no big difference.

> Seriously, I can't see getting so frustrated

If reading documentation and evaluating options frustrates you, you shouldn't be dealing with software development.

> That said, pipe() is not "portable" either, so whether to use platform-dependent code for pipes or semaphores... no big difference.

pipe() is in POSIX base since forever. If it is not portable, I don't know what is. Oh well C is more portable than POSIX, ok; but C does not do IPC at all (actually not even processes) so that's irrelevant. SysV semaphores are in XSI, POSIX semaphore are only in base since recently, previously they were in a Semaphore option.

So yes, pipe() is absolutely more portable. Works on all BSDs, GNU/Linux, embedded Linuxes, Solaris, dead Unixes, Android, QNX, Cygwin (where on the other hand semget seems buggy: https://cygwin.com/ml/cygwin/2019-08/msg00099.html), Mac Os, etc.

Except that now, thanks to the GNU make change in 2017, there are two families of jobserver where previously there was one. There is a family that sets the pipe file descriptors to non-blocking mode, and there is a family that sets the pipe file descriptors to blocking mode.

Alas, the twain do not mix. So anything that wants to interoperate has to know what family of jobserver it is interoperating with. (It has been well known in the Unix world for years that if all programs do not agree on whether a shared file descriptor is in non-blocking mode, difficulties ensue. Yes, nonblock_read() and nonblock_write() would have provided a way to solve this whilst retaining interoperability.)

* https://jdebp.eu/FGA/dont-set-shared-file-descriptors-to-non...

> Oh well C is more portable than POSIX, ok; but C does not do IPC at all (actually not even processes) so that's irrelevant.

C11 adds threads and mutexes

Now just make "-j" the default and all is good :) Why shouldn't make use the available resources by default?

It exposes bugs in your make rules.

You mean not using `-j` exposes bugs? Doubtful. If you are missing a dependency edge in your build graph then there are no guarantees that it will cause an issue with or without `-j`.

The only way to properly solve missing dependencies is to hide targets unless they are explicitly depended on. The only systems I know that do that are Nix and Bazel (and its clones).

If your rules have side-effects that aren't reflected in the dependency part of the declaration but nonetheless depended upon, running the jobs in parallel means the order of execution varies instead of being sequential, so sometimes you may get a file not found on something produced in an earlier rule.

If make knows about all the inputs and outputs of every rule, it can make sure dependencies are built before they are used. The problem occurs when the makefile doesn't tell make about every dependency.

Also the rule scripts themselves can have race conditions. A trivial example is if they end up using the same temp filename when doing work.

Yes I know. But the order of execution can vary even if you don't run in parallel.

For example if you have two targets, A and B, and B depends on A's output but you forgot to say so, then `make` can build B before A even if you don't use the `-j` option.

I thought the behavior I have seen is it's always in the order that the targets are specified.

So if you have:

all: A B

Then if both need a rebuild A is always first, except when -j breaks that.

But I can't say I have played close attention to every time I have typed make.

Ah right, but that's just a lucky coincidence. Make is perfectly within its rights to randomise the order of A and B even if you don't specify `-j`. And if you're driving Make from some other system, e.g. CMake, you really have no idea what order stuff is going to end up in the makefiles anyway.

I prefer processes that expose bugs over those that obscure them...

I've learned over my engineering career to prefer rules which are based on the facts of the situation at hand rather than rules which are applied dogmatically. Dogma is what gave us Chernobyl.

I've had a "make -j" fail due to virtual memory exhaustion (granted, on a 32-bit VM). This was on a C++ project, so it might be less of a problem when building the kernel, but it's definitely not a reasonable default across the board.

I've had similar issues, though it was actually caused by Ninja, and not Make. I was doing a build of clang+llvm itself, and somehow all the link steps ended up happening about the same time. It really exhausted the memory on my system. My system has 32 GiB, of which I rarely go above 10GiB used except for those giant builds.

I really hate it when people do that and don’t provide a way to disable it.

There are a number of python libraries I can’t install on my laptop easily because the native code runs out of physical (4GB!) memory spawning compiler jobs and everything starts thrashing, it’s just dumb to go forking without asking the user first. People who really want it can give make the -j flag.

Bazel is the worst at this. When you try to eg; build tensorflow from source it maxes out all the cores, uses up as much memory as it can, and then crashes when the OOM killer eventually kills it. This is even on a 64GB 64 core machine.

Manually specifying resources doesn't work well either because it uses 2x+ the resources you allocate it.

You can do "taskset --cpu-list 0 make -j"

Memory exhaustion for one. In my experience build systems will happily eat up all of your memory and put you into a swap hell that is difficult to escape from.

Especially painful when one of your coworkers has a beefier machine than you and changes the number of parallel jobs offhand one day, causing your computer to freeze until the OOM killer gets the right build process.

Does this affect select() or only read()? Today if N procs select() on a single pipe, all will be woken up if the pipe becomes readable. Will this now mean that only one will be awoken?

Could the fix have any potential negative effects on other workloads?

Some other build systems use the pipe token approach too, rust's cargo for example.

GNU Make has been fixed, but I wonder if there are other programs which would also have similar bugs in their code? I am reminded of when systemd exposed bugs in system startup scripts, and everybody instead started hating systemd for breaking their systems.

It's not GNU make that got fixed. It's the pipe system call that Linus optimized. Make uses pipe as a synchronization mechanism. So all programs that use pipe this will benefit.

What the parent comment is referring to is a pipe handling bug in early versions of GNU make that prevented kernel optimizations from happening earlier on.

I think this happens a lot with the Linux kernel. Where they struggle to maintain backwards compatibility, in the face of applications relying on undocumented implementation quirks.

I think this case is a bit different, in that the bug already triggers sometimes without this change, the change just makes it happen more often. Which is why the GNU Make bug report predates this improvement.

Suddenly LibreOffice build time’s increase substantially.

What do you mean?

LibreOffice relies extensively on parallel build from Gnu make.

Edit: crap, that autocorrected on my iPhone - that should have been decrease!

Good example of how we develop software these days: instead of fixing GNU make by ripping out that antiquated pipe-based jobserver and replacing it with proper multi-threaded parallelism[1], we rather optimize the operating system to make the antiquated stuff work better.

[1] Having hacked on GNU make I know this won't be easy (it's quite a mess). In fact, it would probably be easier to replace it entirely and fix some of its other deficiencies in the process.

Maybe there is a good reason for that.

There are already dozens of alternatives to GNU Make. Linux already did something similar with Git.

Replacing Make in “your” project is relatively easy. Replacing it in all your dependencies is both a technical pain in the ass and a massive political problem if you want to upstream the change. If you have a lot of dependencies, it’s probably easier just to improve Make. ;)

I would not call avoiding a classic thundering herd problem a bad choice, especially when the top maintainer of Linux does it himself in Linux: which is arguably more probable than him rewriting GNU Make (well we all know he could do it, but this would still be less probable...)

Plus it will not only optimize GNU Make workloads, but potentially other programs that do that. And to be honest, playing with pipe for signaling (and more rarely token counting) is STILL a must-have way to do IPC portably, esp. if you want to integrate with polling on fds. Various OSes have also various new things, but they are non-portable -- maybe someone should try to get eventfd&friends in POSIX?

Anyway, if that's a "good example of how we develop software these days", I would say we managed to develop software the right way at last, making basic stuff work well before succumbing to the tentation to rewrite everything all the time just to play with new shinny techs, getting fresh bugs in the process.

Also you know what has evolved in the last few years, motivating this perf fix? Availability of HIGHLY parallel computers for not too expensive. Yes, I continue to find it highly refreshing that work is done to get all the boring historical software work better in that context. Maybe you think we can afford rewriting all the things all the time to follow supposedly radical technological changes (and using less well designed OSes?), but if so then this thought is not shared widely or at least not followed with action. Rightly so because economically the outcome would be doubtful: just look at the patch, it is way smaller than an hypothetical shiny new rewrite of GNU Make...

Make spawns processes. Using threads would be a terrible idea. fork/execve and pthread_create don't mix well.

The solution is elegant and it has a much bigger impact than just GNU make. There are performance benefits across the board. Maybe the entire system runs 1% faster now. Add a few hundred patches like these and you get a very fast system.

> proper multi-threaded parallelism

So, like buggy and hard to understand?

Reply on news.ycombinator.com