Something to consider: If someone is going to go through the effort to get physical access to a Yubikey, they only need to swap it with one that has a similar level of wear and a similar appearance. At that point, the victim will merely believe that their Yubikey is broken; and/or the attacker will have enough time to use the Yubikey.
For example, I have two Yubikeys. Someone could sneak into my house, swap my spare, and I wouldn't figure it out until I go to use my spare.
Basically: This attack is only "worth it" if your target is so valuable that you can target them in person. At that point, I'd think the target would use something a little more secure than a Yubikey.