Sometimes, you have to.
An example - some code I was working with gave subtly different results on two processors. It turned out this was because one of them implemented FMA - Fused Multiply Add.
https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_op...
It wasn't particularly painful to find this, but if you were attempting to debug this kind of problem without considering lower level issues, you'd spend a whole lot of time banging your head against a wall.