Many folks believe that builds and tests should ideally be "hermetic", and not depend on external state. This can make tests more robust, and facilitates tasks such as bisecting to identify the cause of regressions.
I've also rarely ever touched software that is actually free of nondeterminism, so I am deeply skeptical of caching anything but the simplest test case. And those are fast.