I don’t think it’s the compiler actually. I’m more inclined to blame the project system. In fact visual studios does some hacks just to get around it in some perofrmace sensitive things.
Indeed. It's actually pretty easy to write a custom msbuild logger to instrument (and subsequently chart) all the steps and you discover most of the wall clock time on big solutions with large numbers of projects is not actually compile time. csc itself is fairly competitive with javac. But by Christ, msbuild and nuget restore (which needs to run masses of msbuild machinery these days) are slow.
It's definitely the project system. I once worked on a solution containing ~150 projects. I consolidated all the code down to about 15 projects and the build time of the solution dropped by about 5x. I've been working in Java lately and frankly, I see little difference in compilation speeds between the two.
Rider uses the same project system, so I haven't had a chance to observe any alternative.