Comment by fdr | Hacker News Reader

Comment by fdr | Hacker News Reader

Parent Full thread

On compilation of C programs:

On my unremarkable but not slouchy three-year-old workstation that possesses rotational disks and about 6GB of buffer cache. This cache has not been pre-warmed either, but I can't guarantee it is totally cold. There are with compiler optimizations on, too.

    $ cd ~/codes/postgres
    $ make distclean
    $ make -sj10
    All of PostgreSQL successfully made. Ready to install.
    247.83user 14.22system 0:46.26elapsed 566%CPU

Depending on how you count it, this is 600,000 to 1M LOC.

Let's try complete Linux compile without the drivers, via compiling user-mode linux. This is normally sufficient for debugging file systems or the memory management system (it's really a godsend for that), for example:

    $ cd ~/codes/linux env ARCH=um time make -sj16
    LINK linux
    321.04user 23.83system 1:02.49elapsed 551%CPU

All in all, I have always found the complaints about performance and header files in C programs (note: not C++) either outmoded or unconvincing. There are plenty of legitimate problems -- including tricky issues with semantics -- involving the preprocessor, including something as simple as figuring out when it is safe to remove or reorder a header include! There's no need to imagine problems. This meme about how terrible compile times for large C programs is needs to be put out to pasture, or maybe I just need to be exposed to code bases about ten times the size and feel the need to not use incremental builds.

Thanks for actually posting some numbers, but to me this shows the problem and not the absence of it.

It looks like you have 6 CPUs, and say you have 2e9 cycles per second on each of them. 46s * 6 * 2e9 / 1e6 LOC = 552,000 cycles to compile a single line of code on average. That seems 1-2 orders of magnitude off, no? When you had a 10Mhz computer, did you compile at 20 LOC per second? There's a scaling problem here.

(That's ignoring disk access, but if you did it multiple times I doubt it would go more than twice as fast the second time. And to read 1M LOC from disk cold I'd guess should take on the order of 500ms anyway.)

Is what you're saying is that those compilations are fast enough? You wouldn't want it to take 1 or 3 seconds and see no reason why it should?

Also, Postgres and Linux I'm sure have a LOT better physical structure than your typical industry project. Most industrial projects are bound by talent, I would say. If choosing Go gets rid of this problem, then I would say that's a definite advantage on the side of Go, to be considered when choosing a language for a new project.

And it's true that incremental builds are more important, but on projects I've worked on they take 10-60 seconds for a 1 line change in a .cc file and much worse when you're changing headers.

One correction: assuming the disk access was for files scattered around on the drive, requiring a seek each, you're looking at 10ms per file. At 1,000 files (wild guess) that's 10 seconds.

> Is what you're saying is that those compilations are fast enough? You wouldn't want it to take 1 or 3 seconds and see no reason why it should?

In most practical cases (incremental rebuild) it does take 1-3 seconds. There is just not a huge amount of practical benefit to me. If one has interest in things taking 1-3 seconds for a full rebuild provided one has compiled the code before, there's ccache:

I had to install ccache (reason: this doesn't even cramp my workflow enough to bother until doing this benchmarking) and just do this:

    env PATH=/usr/lib/ccache:$PATH time make -sj10
    All of PostgreSQL successfully made. Ready to install.
    6.05user 1.88system 0:02.48elapsed 319%CPU

If I don't cheat by using ccache, then turning off the optimizer gives me about 50% of my time back:

    120.00user 11.72system 0:21.75elapsed 605%CPU

Here's the result of touching one c file in the executor and doing an incremental build (including linking):

    0.74user 0.18system 0:00.68elapsed 137%CPU

A randomly chosen header file gives me about two seconds, with --enable-depend on:

    3.33user 0.51system 0:01.53elapsed 249%CPU

I like Go, and appreciate that it compiles very quickly in some absolute sense, and for many other reasons, and do not wish for a hideous preprocessor system, but to me claims against the time it takes to compile a reasonably large C program are dubious enough that is can only lead to overzealous suspicion by parties that have to make a quick evaluation on what to spend their time with.

> they take 10-60 seconds for a 1 line change in a .cc file....

.cc is another kettle. Just as the disadvantages of .cc should not be lumped with .c, the opposite also has to be taken into careful consideration: some advantages of .cc are not available to .c, and some advantages of both are retained in .go.

Also, ./configure and Postgres's 'initdb'. Now that's slow, in spite of some efforts to speed up the latter.

For the record, the size of Postgres is at least an order-of-magnitude off from what counts as a "large" C++ program, at least as far as Google is (and many others are) concerned.

Try compiling Chrome (or just WebKit) sometime. It can take well over an hour on a beefy workstation. Incremental compiles are, of course, a lot faster than this, but a long way from "interactive" by any reasonable definition.

But wait, there's more! That bizarre crash or link error you're getting on an incremental compile? Yeah, it's the result of some screwy preprocessor problem that no one's taken the time to debug. You could try and get to the bottom of it yourself, but that would take somewhere between an hour and a couple of days. So instead you suck it up, clean the build output, start up a compile, and go grab lunch. This is the day-to-day reality for WebKit developers, and I don't accept that we can't ultimately do a lot better.

Large server-side C++ programs at Google suffer the same problem, because there's an enormous amount of shared code to compile. It's not as much of a dependency and script/preprocessor rat's nest as WebKit, but it can still get really slow, and you still occasionally have to give up and clean/rebuild when things go inscrutably wrong.

Any assertion that C++ header hell is "good enough" and/or "not that bad" flies in the face of this reality.

Reply on news.ycombinator.com