The amount of effort and extra hardware required to maintain this reproducibility across different hardware (cpu generations, GPU hardware/drivers, etc) feels insane to me. For example, people seem extremely interested in performance, but insist on using the much slower modes in MKL to maintain the bottom few bits of their golden data.

Do you find this genuinely important? If not, have you had any success encouraging e.g. validating against golden data in a non-bitwise comparison?