Not bashing at all (you are somewhat in a better place than my own organization) but a point could be made that you could break the code down into subprojects and manage dependencies.
A lot can be done to reduce your typical or average build times (make everything data driven so you don't even have to rebuild the code!), but C++ makes it very hard to keep your worst-case build times in check. And I tend to be one of the guys tweaking and fixing things that invoke those worst-case build times. A simple example: Annotating logging macros to catch printf-style format string errors at build time instead of crashing at runtime if you're lucky.
Properly testing involves rebuilding most source files, on at least 3 build flavors to test against MSVC, Clang, and GCC...
I'd enjoy working on this :) It looks like a nice application of true build pipelines using GoCD. If you're interested to discuss this offline, do let me know.