The C++ Build Process Explained
github.com
github.com
In C and C++ parlance "definition" should be "declaration" and "implementation" should be "definition".[1] The terminology is important if you don't want to get confused when learning more about C and C++. This is compounded by the fact that some languages describe these roles in the author's original terms. (Perhaps the author's terminology reflects his own confusion in this regard?)
[1] This is indisputable given the surrounding context, but I didn't want to paste 3-4 whole paragraphs.
In extremely old code, I sometimes see people preferring to manually write declarations of functions (even libc functions) in every source file instead of including a header.
To add to the confusion, in certain cases, it is possible to put a function's definition in a header file (for example if it's a function template, or in an anonymous namespace, or the static keyword is used to indicate internal linkage). So it is possibly to write this function definition manually in every translation unit.
Otherwise, the ODR rule requires functions to be defined exactly once.
> One and only one definition of every non-inline function or variable that is odr-used is required to appear in the entire program (including any standard and user-defined libraries). The compiler is not required to diagnose this violation, but the behavior of the program that violates it is undefined.
This sounds like it would be easy – just put the definition a header file. But even if the text of a function is identical in different translation units, it can still be ODR-different between them if the symbols that they look up are different due to other header files included before them declaring different thing or doing things like "using namespace std". Argument-dependent lookup is especially dangerous here. As with other ODR violations, this causes undefined behaviour and compilers/linkers aren't required to issue a diagnostic (and they usually don't!). I believe C++20 modules will solve this problem.
-fsanitize=undefined with LTO has shown ODR violations for a long time
Exactly. This is why you should never put anonymous namespaces and definitions of objects with static linkage in an header file.
In Titus Winters' recent Pacific++ talk[1], he pointed out that even something as simple as including an 'assert' statement will violate the ODR if some compilation units are compiled with 'debug' settings and some aren't. This can easily happen with build systems that cache compiled object files if changes in compilations flags don't automatically invalidate the cache.
As berti said: enums, constants, macros, simple struct type declarations, typedefs, and templates, are examples of hand-written code that might belong entirely in the header file.
It would be possible to keep these constructs in a hand-written header file while auto-generating the rest, but C++ developers aren't convinced this is worth doing.
C++ has been working on modules to fix this for years now. It turns out to be harder than you would think. Everyone agrees with the basic problems statement, but there is disagreement on the details. All sides have good points in favor of their approach, and there are some places where you have both as they are incompatible. Progress is being made (and a lot of the incompatibilities turned out to be solvable but it took years of thinking to come up with how)
Have you checked out the standard library's header files? If you do, you'll see they are full of templates, and constexpr functions. In some sense, the header file is where most of the code is.
Have you also checked out the concept of a single-header dependency? Due to the myriad different ways to manage dependencies, those dependencies that consist of just a header file is extremely convenient to have. This is the logical conclusion when you can put increasingly complicated things in header files.
I will correct this as soon as I have some time to do so.
Prelinker:
1. Compile each file, noting which templates are needed in a section in the object file
2. Have a special program called the "prelinker" run before the linker that reads each .o file, and then somehow instantiates each template (which usually, but not always requires reparsing the C++ file)
Weak Symbols:
1. When compiling instantiate every needed template, but mark it somehow in the object file as being weak, so that the linker only pulls in one instantiation for each definition.
The prelinker used to be more popular, as if you e.g. instantiate the same template in every single file, your compiler does tons more work with the weak symbols approach, but now weak symbols are popular both because they are much simpler to implement and the fact that compilation is usually parallel, while linker is typically not means that walk clock times may even be faster.
People started to use templates for metaprogramming and from that point on the scope for "reuse" of templates isn't really there. (Reusing parsing might be plausible, but it's really difficult because parsing is extremely context-sensitive because of SFINAE, #defines, etc.)
Some might comment that "modules" is "export template" all over again, but this time there are actually 2-3 implementations of 2-3 of the proposals and everyone is confident that the remaining minor problems can be resolved satisfactorily... and they're all exchanging experiences to help each other!
In compilers other than Visual Studio, yes (or so I'm mostly told - I don't have all that much experience with them), but msvc has had them since at least VS6 (late 1990's) when I first started using them, and they work very well and have saved me many, many hours since then. Maybe once or twice I had to delete the pch file in all that time, and that was most likely more of a issue of the GUI mangling the saved internal state than the actual compiler.
I've heard pushback against precompiled headers from Unix land for 2 decades, I'm not really sure where it comes from. I have the impression it's mostly cognitive dissonance - 'msvc has it and gcc doesn't, therefore it must be bad because gcc is 'better' than msvc'. It's similar to #pragma once - in use, it's objectively better in every possible way than include guards are, and gcc fanboys still dismissed it back when gcc didn't have them.
There is a reason pragma once is not standardized. Defining when to include lines refer to the same file is extremely hard.
https://itanium-cxx-abi.github.io/cxx-abi/abi.html#linkage
[Edit]
EDG has a customer list here, but not all customers are compiler makers; frontends are useful for analysis too.
[0] https://gcc.gnu.org/onlinedocs/gcc-5.5.0/gcc/Template-Instan...
Cfront also basically never worked well.
1: the technical term is "compilation unit" since #include means you're never compiling just one file
Split the template into declarations and definitions, similar to .h versus .cc.
Then #include the definition into some module where you write template instantiations for all the needed types.
I suppose at the linker layer maybe but even that is just once globally for the program. Right? The OOP God's demand polymorphism!
If memory serves me, the C++ version is a class with all empty virtual functions.
Those 2 plus other crazy rules (like only one interface defined per file) mean that many projects will end up with just as many interface files as a c++ project would header files.
But still, no IDE or tool I've seen does this better. A header file gives you a very nice overview, and it really isn't any hassle to speak of to keep them up to date.
It takes getting used to (as with everything when starting out with a new language), but before you know it you might even start to miss them in other languages.
Dear lord, all those broken builds I've seen over the years would love to talk to you about that.
They're also bad as an overview. They may have been good in the 1980s, but nowadays a proper documentation generator gives you better formatting, search features, and cross-references. Inline documentation is particularly obnoxious in header files: I like seeing function-level documentation alongside interface and implementation, but if I put the documentation in both the .cpp and the .h file it's duplicated in two places and easily gets out of date.
I have the feeling that the difference matters to almost nobody and should be avoidable by the few people it actually affects. I had more issues with colliding include guards than I had with symlinks and I still end up replacing CLASSNAME_H with PROJECT_CLASSNAME_H in our own headers every now and then since the autogenerated guards are too naive.
Include order is a valid complaint, the others truly are non issues.
I mean, we have other more pressing issues. Such as the fact that garbage collection is still a thing in 2018.
> ...but nowadays a proper documentation generator gives you better...
Examples of that?
> I like seeing function-level documentation alongside interface and implementation, but if I put the documentation in both the .cpp and the .h file it's duplicated in two places and easily gets out of date.
This is the only thing that I dislike about header files. And it is a constant reminder how bad IDEs are at handling this.
I hope he meant something else since he said "nowadays". Doxygen is ok if you haven't setup your build environment. But for when you actually are in the code (and somewhat familiar with it) I very much prefer just reading the header files than switching to a browser.
This happens because there is some redundancy between the header file and the source. You can often arrange your code so that there is no redundancy (or almost zero).
how so ? every IDE I've used automatically changes the header if you change the cpp and conversely
Modula-2 does it better. Each module split in two files - definition and implementation. Unlike C++ header files, def files are compiled rather that inlined, so they don't have a potential to produce different result for each source file that references them. The result is 100% reliable dependency graph across all source files, no need for a makefile, and blazingly fast compilation (each def file is only parsed and compiled once).
[1] https://discuss.ocaml.org/t/what-is-the-reason-of-separation...
Why aren't Delphi, Oberon, and Modula-3 immensely better? Each file is an independent module except the one with main. Then the compiler quickly figures out in one pass what all the function signatures are. The minor drawback with this is that the one-pass method requires you to pay attention to declaration order, though a two-pass method renders that approach irrelevant. The IDE has all the symbolic goodness you need, and no goddamn header files. Also it's still easier than dealing with managing the dependency graph yourself as is necessary with C file include order.
I say this as a guy who enjoys C and has done way more work in C than Delphi, but give credit where it's due.
"Uses BGI;" and I'm done :)
Thankfully there is Bitsavers.
"Circular Unit References" chapter on the Turbo Pascal 5.0 programmer's user guide.
https://archive.org/details/bitsavers_borlandturVersion5.0Us...
But they don't.
Also, a header file will hopefully be logically structured by a human. Something generated by the IDE will not.
You could even make them part of the build, if you want a continuous up to date version.
That makes a huge difference.
Headers (or at least 'separate specification of API') are/is great, and I miss it dearly in other languages.
Already in Turbo Pascal for MS-DOS there was a TPU dump tool, which would generate a "header file" from a TP module.
Header files are not a way to only expose the interface. You give up a lot more in C++.
I've never had to deal with pesky header files until I started developing C and it immediately struct me as a royal pain in the ass. Even after couple of years of developing C/C++, I find the whole concept of header files archaic. Include preprocessor directive literally copy pastes stuff with no intelligence what-so-ever. The user is now burdened to ensure #includes are guarded to what I call a patchy half-baked hacked up solution - #IFDEF/#DEFINE/#ENDIF and #pragma in C++.
It should be handled automatically by the compiler/preprocessor or IDE and I believe it is now being addressed in the C++17/20 spec with the advent of "modules". This thing should have been written up way back in 1989.
Sorry, but it is not "completely false". Doing it properly requires a carefully designed interface to hide internal data structures, and splitting out the end user headers from the internal headers, but it works.
> I've never had to deal with pesky header files until I started developing C and it immediately struct me as a royal pain in the ass.
It's like saying, "I've never had to deal with pesky .py files until I started developing Python."
Out of curiosity, how? (If pointers or other mechanisms for memory indirection are allowed then it's pretty easy, so let's agree to ban those.)
Not a rhetorical question, despite the parenthetical.
Here's one popular way to do it in C++: https://en.cppreference.com/w/cpp/language/pimpl
I guess my tongue in cheek response to your parenthetical would be to ban the use of hidden internal data structures ;-)
Doing the same with objects require pointers to hide the details of any private members. A common pattern is called Pointer to IMPLemenation (PIMPL).
Not sure why we need to exclude pointers?
In c++14 this is very much simplified with the use of unique pointers.
std::unique_ptr<Foo> make_foo(...);
When the parent (er, great grandparent?) says their parent is wrong to deny that headers only reveal the interface, I think it's a little disingenuous to base that on an unrevealed assumption that PImpl is in play. PImpl is basically the the idiom that begot Java. One reason I might opt for C++ over Java, is a desire for finer control over the location of memory -- but it's important for me to know that to actually get that, I'll probably have to sacrifice information hiding. It's a trade-off. Yes, on some level it's better to have the option to make that trade-off, but the product here is "encapsulation or value semantics", not "encapsulation and value semantics".
Mind, I'm not a C++ developer. Maybe these days link-time heroic optimization makes the "right" decisions and collapses these kinds of indirections in all the sorts of situations you'd want it to. I write a lot more Java, and my understanding is that HotSpot gets up to a lot of heroics pertaining to this stuff these days -- I've noticed HotSpot will churn through a workload involving processing a collection of records far more quickly if you can arrange for it to stream through an array, even if you'd expect the records to be scattered randomly throughout memory.
Conceptually this should be possible with some preprocessor/header tinkering; something like:
private.h:
struct Bar {
double x;
double y;
};
#define BAR_DEFINED
#ifndef BAR_DEFINED
struct Bar {
char opaque[16];
};
#endif
struct Foo {
Bar bar;
};
#include "public.h"
#include "private.h"
#include "public.h"However I'd say it's an ugly hack which should only ever be used if you _really_ need the performance.
if you want to put the object on the stack, the compiler has to know the size of the object to reserve enough space on the stack. How can it know the size of the object if it does not have its full definition somewhere ?
Some languages have, like ADA I think, have first class support for runtime sized, stack allocated types, so it might work there.
Sometimes you need to. You cannot entirely stack allocate an object that uses PIMPL. Also you cannot allocate an array of such objects compactly in memory.
On the other hand, if you want to be able to evolve the class member variables but still maintain a stable ABI, you need to hide the memory layout, for example with PIMPL. But this is a C++ limitation. For example Objective-C* (and also soon Swift) allows modifying the class layout, adding properties etc, without changing the ABI.
https://en.wikipedia.org/wiki/Objective-C#Non-fragile_instan...
You actually can, if I'm not misunderstanding: http://www.gotw.ca/gotw/028.htm
so it's just syntax sugar for PIMPL.
In Objective-C 2 the object meta-data contains a table of instance variable offsets. The dynamic linker can modify this table at load time so you can freely add both instance variables and methods to new revisions of a class.
So what is the deal? Well, when the holder object itself is heap allocated, pimpl is inefficient because every access will require dereferencing two pointers. Also you cannot put protected or virtual members in the internal pimpl class (then there would be no point to have those in the first place).
That being said, it is not like Objective-C is some pinnacle of performance - you cannot allocate objects on the heap, and the compiler doesn't perform any devirtualization. So for performance critical code you have to drop down to C or...C++ :)
If you don't want to expose class internals, just use the PIMPL idiom. It's an extra indirection to protect your own abstraction, so naturally C++ decides to opt for performance by default.
sorry what ? most C libraries I know hide their implementations behind opaque types such as `typedef void* my_handle_t`.
struct my_type;
struct my_type *my_type_new(void);
You can also do this in C++ of course, but you can't use member functions this way. My real complaint here is that the Pimpl idiom in C++ is more cumbersome than a simple forward declaration and free functions, which is available in C.
Well, it's easy to reason about them. The thinking goes about the translation units.
The real problem is that the actual interface is not enforceable: the symbols in the binaries are just names.
100% agree. That said, that doesn't mean that they're the only way to do it, or even the best way.
The return value of the `add` function, in most ABI definitions, would be stored in a register. After that, the `main` function may then copy that value to its own space it has reserved on the stack.
This is at odds with the description in the article, which seems to describe `add` passing its return value to `main` via the stack.
(This is assuming no optimizations - all this would most likely be inlined anyway, with no function call.)
The special thing about them though is that the linker will not include any .o files for which no symbols are referenced.
You can think of the typical linker algorithm as follows:
If I am missing a symbol X scan through all not used objects in each library until you find it, then include that .o file. Now check if any symbols are missing again and repeat until done.
That properly describes linking using --start-group/--end-group. Without those flags, the process is closer to "look in the first .a file for definitions of all currently undefined symbols; then look in the next .a file for definitions of remaining undefined symbols; etc.". The difference becomes apparent if you link chains of libraries in the wrong order, or if you have cyclic dependencies between libraries; normally they will not be resolved unless you use the grouping flags. (But really, you should avoid making such cycles in the first place!)
That's a very simplistic and error-prone description of what a library is supposed to be. Even in high-level descriptions, it's very important to be aware of the fundamental differences between static and dynamic libraries, and how they are integrated in the build process, which has a fundamental role in basic tasks such as deploying applications.
Static libraries are more or less that yes (at least, that mental model is sufficient for pretty much all day to day use of them), but dynamic libraries aren't, at all.
As an aside, I found it very weird that the OP claimed that 'nobody uses static linking'. Wut? Static linking is everywhere (so is dynamic linking, of course, but implying that one is merely a quaint remainder from the past is just so odd).
I don't hate dynamic linking as much as I used to, but it is still a peeve of mine. Linux is one of the most backwards compatible kernels, but the heavy usage of dynamic linking means that a linux program from 20 years ago either won't work at all, or will be buggy.
A statically linked program from 20 years ago will work (but possibly won't have sound; though it's possible to get either alsa or pulse audio to emulate OSS and then even sound will work).
> a bundle of .o files [note ".o"]
> e.g. [...] libpython2.7.a [note ".a"]
adds, rather than resolves, confusion.
.a is an extension for one type of library, sorry if that confused you, but it can be ignored for the purposes of this explanation.
I'm busy for the next few days but this is near the top of my TODO.
Thanks for the feedback :-)
Check out this HN discussion on an ebook on linking and loading. The book itself is a treat, and the discussion around the book is very informative as well.
I almost sort of get what the author means here but then I don't really. I mean, there is no 'state' for the compiler that is modified by precompiler directives, so this is probably an analogy or simplification he's making here, but I don't really understand how he gets to the mental image of 'compiler state'. Why not just say it like it is: the preprocessor generates long-assed .i (or whatever) 'files' in memory before the actual compiler compiles them, the content of which can be different between compilation units, because preprocessor preconditions might vary between compilation units?
What you propose as a replacement (an in-memory file) does not provide any insight into why the same file preprocessed twice may end up looking different or why order of included files matter.
When you take the preprocessed state of a compilation unit, by having the preprocessor write it out to disk, and show someone what the effects are of passing one or the other -D flag, or change the order of includes - that directly and concretely shows what is going on. And then this preprocessed file is passed on to the actual compiler. There is a clear separation between stages, easy to understand, and useful to boot when the time comes you have to debug an issue related to it and you want to look at the preprocessed file to see what's going.
Yes, I absolutely agree. And to be honest I cannot imagine explaining how preprocessor works without describing it as a separate entity.
In fact, ideally you'd even generate all binaries for a project in one go but that may be taking it a step too far :-)
At any rate, I searched google scholar for any experience reports on this and found nothing. It must have a more technical name I guess...
It's used sometimes, I think mostly to help the compiler optimise better.
Build times for a full rebuild may be faster, but may not, since traditional builds can use many CPU cores. However, it stops incremental builds from working - if you modify one source file, you have to recompile everything.
It might still be interesting to read through this stuff--throw "tromey gcc compile server" at a search engine and see what comes up.
Compilation of the 11Mb object file takes about 30s and requires at most 1GB memory (the lib is not that huge, 20kloc), but I think when we started on it, the build took a few minutes for the lib alone. So we save some dev time, but building all unit test executables still takes an additional 1m30s. So that's only a minor improvement. But I think the real gain is a much better optimization (the architecture of the lib is great to maintain and bugs are at least critical or might even have a lethal impact; there is a lot of potential for inlining/LTO).
VC++ calls that thing “Link-time Code Generation”: https://docs.microsoft.com/en-us/cpp/build/reference/ltcg-li...
LLVM calls it “Link Time Optimization”, pretty similar: http://llvm.org/docs/LinkTimeOptimization.html
There are too much Interests and the standardization would kill many of them