Things change with -Ox, but then again, who really cares? Generating optimized binaries is not something I do routinely, except when tuning for performance.
When I'm generating a binary for release, I couldn't care less if the project actually took 2x times to compile. In fact, to use feedback-directed optimization (which on some projects may make a marked difference, usually larger then the generated binary at O3 between gcc/clang), you actually have to build it twice, and _very_ few people actually bother.
clang used to be much faster at building, but then again the binary was never equally on-par performance wise. As clang is getting better at optimizing, the compilation times are getting more and more similar.
This is again, all hardly surprising for me.
I _have_ worked on projects using C++ where build times where incredibly slow. They all shared the common attribute of #include <megaheader_with_all_class_definitions> and "using namespace library".
For other projects, as long as rebuilding an object takes 2 seconds or 10, does it really make a difference? In those cases, linking actually takes disproportionately longer, to the point that compilation times are insignificant.
It's a genuine question.