Hacker News Reader

Comment by symisc_devel | Hacker News Reader

The technique is known as Amalgamation and was first introduced in the SQLite2 source tree. In my company were we do embedded software[1], this shown to be a powerful technique. Not only, the compilation is extremely fast (A 1.3 megabytes of C code, took 9 seconds to compile on a core I3), but a modern compiler will be able to do additional optimizations on code when it is contained with in a single translation unit.

[1]: https://unqlite.org, http://ph7.symisc.net

View on HN

Why don't compilers do amalgamation themselves as a pre-compilation step?

The compiler runs too early. If you're compiling C or C++, each individual .c/.cc/.cpp file gets compiled into a .o/.obj file by the compiler. Once these are all built, the linker combines the .o/.obj files and produces a library or executable. Compilation is finished before you get a chance to combine anything—the linker is what combines the code, and it can't do much optimization.

However, with LTO (link-time optimization), the compiler doesn't finish compiling and writes out .o/.obj files with partially processed outputs. The linker is modified to re-invoke the compiler to finish compiling all the files at once. In GCC this is available with -flto.

Actual amalgamation, where you combine many C files into one, is often not possible without modifying the files. It works for SQLite because they've made sure that their code works with amalgamation.

In practice, LTO compiled .o/.obj files just contains compiler IR. Combining C/C++ files into one is harder due to things like name resolution, etc.

My guess is that it's to prevent compilation of code which is unused. If code which isn't actually called anywhere is amalgamated and compiled, compilation time could actually increase instead of decrease.

Seems easily avoidable though through basic dependency analysis, though.

That's not actually true. You have to pass special linker flags to tell the linker to avoid including code which isn't used. With GNU Binutils and GCC, those flags are -Wl,--as-needed and -Wl,--gc-sections. Normally, unused code is only excluded if it is part of a static library. Even then it is only excluded or included an entire file at a time, unless you split files into multiple sections with the compiler (which has drawbacks—the compiler can do certain optimizations if it knows that two pieces of code or some code and data end up in the same section).

Ah, good to know – thanks for correcting.

Because it means rebuilding everything every time, rather than just the relevant .o and re-linking.

On a large enough codebase, being able to rebuild only what you touch and relink is the only way to get acceptable build times in dev.

You could argue that's the point of link time optimization.

How is this different from so-called unity builds?

Reply on news.ycombinator.com