In our case, we built programs that ran enormous numbers of semi-random programs on the accelerator and compared it to reliable results computed offline. About 1 in 1000 chips would - reproducibly - fail certain operations. Identifying this helped solve problems many of our researchers reported on specific accelerator clusters- they would get a Nan in their gradients which would kill training, and it was almost always explainable by a single processor (out of ~thousands) occasionally corrupting a float.
At a previous company we had to run a dll in our web servers (provided by the payment processing company) for PCI compliance reasons, and we later discovered it was messing with our FP flags and as a result serialization code was producing invalid floats. That was a fun one.