My experience is the exact opposite. Moving a multimillion line C++ code base from msbuild to CMake/ninja on Windows cut the build time in half.
Chrome got even better speedups I believe by building with clang/ninja on Windows.
Bazel is where the real benefits lie by reusing other people's (or CI machine's) partial build artifacts via a centralized cache and by avoiding to run tests that are not affected by code changes.