One of the main metric for build time for me is to make it fast enough so you do not want switch to other task at all
I always see it as a hierarchy; editor, local runtime, pre-commit / pre-push hooks, CI. At the moment, if this setup works, my pre-push hook should only run various tools (linters, unit tests) on the files that were affected by the commit. I'll leave it to CI to run the full suite of verifications and end-to-end tests.
My CI is basically 5% git checkout (varies), 2% build (a few minutes), 80% run tests, 5% static analysis, the rest is misc crap like uploads of artifacts, code signing, etc.