Every bit of nondeterminism in your binaries, even if it's just memory layout alone, might alter the behavior, i.e. break things on some builds, which is just really not desirable.
Why would you ever want builds from the same source to have potentially different performance, different output size or otherwise different behavior?
IMO tivoization is completely unrelated, because the vendor most certainly does not need reproducible builds in order to lock down a platform.