Note that if you want to go a bit further, you should make your own minimal build system that replaces CMake. That alone will make your project compile much faster!
I've made my own in Python that generates Ninja files only - and it's surprisingly not that much work (especially if Claude Code can help you).