Hacker News Reader

Comment by batou | Hacker News Reader

I think something that requires 8Gb memory to run the linker has a bug.

The memory use at the "link step" is so high because due to whole-program optimization, that's the point where everything actually compiles.

It used to be fairly typical for large C++ applications. Chromium and Firefox had problems with it before GCC got fixed. Sounds like clang still has work to do there.

I completely agree but my point is more that the compiler shouldn't be that large even whilst linking and doing whole program optimisation. Proportionately to the size of the output binary the overhead is unreasonably large. Firefox and Chrome for example are problematic due to the architecture. I've built modular applications which compile smaller units (.so's) that are runtime linked using dlopen() etc. Not quite the same scale but this was in a time when the kit was a lot slower.

I'd like to see a return to a simpler time. We can afford to lose some of the optimisations now and produce clean, simple, fast and predictable compilers.

I think Roslyn/RyuJIT on the .Net platform is an example of where this all falls apart: bad tail call optimisation leads to non-determinism in parameter passing. If the compiler consists of lots of code then there are lots of bugs and you can't afford bugs there.

Can I interest you in the original Plan 9 compilers? Their source should be around somewhere and they don't do much in the way of optimization, if any, so they shouldn't any giant memory use problems.

You can indeed. Are they the ones that shipped with the original Go as well?

My understanding is that the original C toolchain in Go (only recently removed in 1.5) was based heavily on the Plan 9 compilers.

In fact there was a rather inflammatory blog post[1] whose basic theme was that "Golang is trash" because the toolchain was so simple. One of the main Go developers then responded[2] on HN. Different people value different things I suppose.

1. http://dtrace.org/blogs/wesolows/2014/12/29/golang-is-trash/

2. https://news.ycombinator.com/item?id=8817990

I've hacked heavily with both; I've even compiled the Plan 9 kernel using Go's compiler+linker. The compilers are damn good. I went from no knowledge of 6[cl] (the amd64 compiler&linker) to pretty comfortable with the source in about 2 days.

Go uses customized but recognizable derivatives of the Plan 9 compilers. Go's compilers are really only intended to compile the Go tools, so you might find it painful to build anything else.

Getting 6c/6l/6a compiled on Linux would be easier these days because gcc's -fplan9-extensions flag will bring you pretty close... check out https://github.com/rminnich/NxM/tree/master/util for the source and scripts to build them on Linux. I believe you can make it output elf binaries but I've never tried using them to build a Linux program.

If you want to greatly speed up compilation at the cost of some runtime performance, just turn LTO off. I've never heard of a project requiring it to function.

Disclaimer: I'm trying to compile llvm/clang on OpenBSD with gcc 4.9 and binutils 2.17.

Linking several hundred objects in a dozen or so libaries out of 2.5kish objects together is going to get hairily expensive memory wise no matter what way you do it.

Compiling with debug information embedded raises that to around 12GB to do it in a practical timeframe, though it may have dropped recently with some tweaks that have been published on llvmdev. Unless you've willing to wait a few hours.

GCC is no slouch in that area either but its usage spikes feel lower.

(Source: compiler dev)

Reply on news.ycombinator.com