Hacker News Reader

Comment by psi-squared | Hacker News Reader

If I've read this correctly, the 'sem' mode lets you submit several lots of jobs with an overall limit on the total number of tasks running at a time (rather than one limit per lot of jobs).

That on its own is super useful for what I'm working on right now. But what would make it even more useful, is: can you get GNU make to use 'sem' instead of its own jobserver? That way I could run almost everything I need to under one overall task limit, and that would be really nice to have.

(For this reason, I'm a fan of the idea that every program with its own 'parallel execution' mode should be able to interact with a common jobserver. The 'make' jobserver is, as far as I know, the simplest, and should be pretty easy to support: http://make.mad-scientist.net/papers/jobserver-implementatio... )

View on HN

I don't think you can use a different job server for make, all you can do is have a make rule that launches parallel or something else, but then you lose dependency tracking on the sub-tasks.

Are you running parallel make tasks where each task is also doing something multi-threaded or parallel? Like using make -j 8 won't work for you?

Make does have the -l load average task limiter when but I've never gotten it to work reliably, it always starts way too many jobs at first and chokes for a while before calming down. Often that won't work for me, but maybe it will help you?

My current workflow has a lot of "Run make -j<lots> to build, followed by parallel -j<lots> to run all the tests", but sometimes I want to compare/test multiple different versions of the code (in a way which, sadly, doesn't work well with incremental builds). In that case it'd be nice to be able to just spin everything up and not have to worry about overloading/under-loading the machine I'm working on.

I know what you mean about the load average limiter - parallel behaves like that too. I think the --delay option to parallel is supposed to solve that (I haven't tried it - will try tomorrow), but I don't know if make has anything similar.

Finally, on further reading, it definitely seems technically possible, even if it hasn't been done so far. The make documentation has a section on the jobserver protocol, which looks complete enough to write both the client and server parts: https://www.gnu.org/software/make/manual/html_node/Job-Slots...

So if nothing exists so far, it's something I might look into writing myself.

> Finally, on further reading, it definitely seems technically possible https://www.gnu.org/software/make/manual/html_node/Job-Slots.... So if nothing exists so far, it's something I might look into writing myself.

Holy moly, that's kinda crazy, but would be fun, you should totally do it! Looking forward to your blog post! ;)

Reply on news.ycombinator.com