GCC Automatic Parallel Compilation Viability Results Help Up to 3.3x

I agree. In all of the large projects I have worked on the build systems were constantly optimized under the observation is that compilation is infinitely parallelizable and the latency doesn't matter nearly as much as throughput but the link is the expensive step on the critical path.

Linking improvements would have far-reaching benefits. I have similar observations to your comment regarding both C++ and rust.

I'm not familiar with clang's support for parallel linking, so maybe this isn't what you mean, but GCC supports parallel ltrans for LTO, and IIRC GCC 10 made multithreaded ltrans the default behavior.

Try building the SQLite3 amalgamation.

For C++ it really does become an issue, since each individual file can take over a second to compile (usually depending on how heavy the template usage is), even for moderately-sized files (~1000 lines).

Projects using templates and heavy metaprogramming often see most of the compile time spent in template instantiation, not in link time. For this kind of projects I think this would help.

Lets certainly hope clang will eventually manage to improve things in that context, because the restriction against parallelization during the optimization step when there is a single compilation unit is quite punishing.

Doesn't clang already implement this ?

LLVM can process multiple codegen units in parallel already, and many frontends split TUs into multiple codegen units.

No.

LLVM is not amenable to using multiple threads to compile a single TU. The use-lists of global values (such as functions, global variables, or constant expressions) include all uses from all functions, so parallelizing on a per-function basis requires acquiring locks (or some sort of lock-free data structure) to add, remove, or iterate these lists, which would considerable overhead on a relatively common operation.

If you want more details, you can read the recent thread on llvm-dev discussing this: http://lists.llvm.org/pipermail/llvm-dev/2020-March/139606.h...

Such a compiler exists, it's called zapcc (was a fork of clang 3.something). Sadly it has been abandoned and never merged even though it was opensourced.

We already had function level compilers back in the early days, Energize C++ and VA for C++ v4, but they were too resource hungry for what companies were willing to pay for and they died.

There is an Energize C++ demo floating around on YouTube.

does ccache or sccache help?

I rarely touch C / C++, but when I do I've been having to git bisect stuff. Having ccache in between has been invaluable in reducing the run time.

I suppose it's more automatic than manually splitting up your source to facilitate parallel compilation, but I agree, it's not very clear.

Perhaps it would have been better to describe it as intra-process parallel compilation in GCC, or something along those lines.

Imagine you have one source file with 100 of functions which you want to compile. Traditionally it could be compiled only using one core, processing function after function. Assuming we are processing already preprocessed information, there's no theoretical reason why more cores couldn't be used to compile the functions: e.g. with 4 cores, each could get 25 functions. The practical reason why not is not only that it increases code complexity in the trivial case (all the functions take the same time) but that in non-trivial case you can end up trying to do a lot of what the OS and make system together do with separate processes doing the separate compilation of many files, dynamically deciding what to do at which point.

This experiment tries to make a case that something in that direction is doable and worth doing. I missed the good proof for the later, based on the analysis of state of the existing projects, however. I would personally rather consider the completely opposite direction:

In the big C++ projects, most of the functions are already present in small compilation units, and the total of the build process mostly spends time in processing the same huge set of headers for every compilation unit. Often, the whole process would be faster if more compilation units would be compiled as one(!) when the headers aren't used in a way that the semantic of them changes depending on the way in which they are used (I've heard that, of all, LLVM actually does that unfortunate thing, but I haven't spend time analyzing that myself).

Google Summer of Code participants are not interns, they are paid a stipend to work on open source projects. And–isn't it good if an intern project ends up being used?