I completely agree but my point is more that the compiler shouldn't be that large even whilst linking and doing whole program optimisation. Proportionately to the size of the output binary the overhead is unreasonably large. Firefox and Chrome for example are problematic due to the architecture. I've built modular applications which compile smaller units (.so's) that are runtime linked using dlopen() etc. Not quite the same scale but this was in a time when the kit was a lot slower.
I'd like to see a return to a simpler time. We can afford to lose some of the optimisations now and produce clean, simple, fast and predictable compilers.
I think Roslyn/RyuJIT on the .Net platform is an example of where this all falls apart: bad tail call optimisation leads to non-determinism in parameter passing. If the compiler consists of lots of code then there are lots of bugs and you can't afford bugs there.