Are both cases compiling to the same target architecture? If not, you may well be comparing the relative performance of different compiler backends instead of comparing the performance of your CPUs.
(+ now I see it's rust: how parallel is your build, really?)