Parallel is running 8 git clone jobs at once (as asked for) and each git clone is starting as many index-pack threads as it wants. Temporarily setting pack.threads to 1 (via git config) would help here.
Parallel is running 8 git clone jobs at once (as asked for) and each git clone is starting as many index-pack threads as it wants. Temporarily setting pack.threads to 1 (via git config) would help here.
The solution is for only one of the two layers to parallelise.
—⁂—
Given that you talked about pack.threads, here’s its description from `man git-config`:
> Specifies the number of threads to spawn when searching for best delta matches. This requires that git-pack-objects(1) be compiled with pthreads otherwise this option is ignored with a warning. This is meant to reduce packing time on multiprocessor machines. The required amount of memory for the delta search window is however multiplied by the number of threads. Specifying 0 will cause Git to auto-detect the number of CPUs and set the number of threads accordingly.
If you have a common scheduling API, you can manage this much more elegantly. For example make(1) can control the concurrency level across recursive invocations by using a "job server" https://www.gnu.org/software/make/manual/html_node/Job-Slots...
With this, you can have, for example, 3 top level subprocesses, each spawning multiple threads of their own, but never exceeding the CPU count.
Alternatively, parallel could make it's subprocesses think that they're running on a 1-core machine, although this may have some subtle side affects.
Or just use `git -c pack.threads=1 clone`: https://git-scm.com/docs/git#Documentation/git.txt--cltnameg...