It is always faster in all cases from my experience. It should not, but it is. Try.
The best possibility is if your build system can detect which files have changed and exclude them from unity builds, as this gives you the "slow path" the first time you change a file, but from then on you get the fast path behaviour.