If you have a Yubi Key and you are worried, read this part from their report:

> the adversary needs to open the device (...) Then the device must be re-packaged and quietly returned to the legitimate user (...) assuming one can find a reliable way to open and close the device, we did not work on this part

Yubi Keys are supposed to be tamper-evident and you can also put tamper-evident stickers on them. I am more concerned that a determined attacker may eventually find a way to record the signals without having to remove the casing.