We've encountered similar constraints. Part of the solution is adopting a build system that can aggressively cache and parallelize.

In our case I wrote a light-weight version of Bazel: https://grog.build/