You will find that even though GCC spends 80% in the optimizing stage (I won't bother to refute your claim), impact of this is negligible in the grand scheme of things. The build is slow is because it scales O(n^2) with the number of files, and because there are many more steps involved in building a product, not because the compiler is slow (although GCC is).
Go is designed to handle dependencies between objects in such a way so that it scales O(n).
The Solaris/OpenSolaris/illumos build builds with two compilers at the same time. The Sun/Oracle proprietary compiler that generates better code, and GCC. GCC generated binaries are not usually used, instead GCC is used a shadow compiler to catch potentially not portable statements.
You will find that if you disable GCC and build with only one compiler, build time decreases by 2%, not 50% as you might have expected.