An attacker not only needs your username and password, they also need physical access to your key. They then have to disassemble the device. If they want to give it back to you, they'll need to reassemble it.
So not exactly trivial!
A blob of nail-varnish over a plastic seam might be a useful canary.
But this does highlight one weakness of these FIDO tokens - you have to manually maintain a list of where you've registered them. And if your token is lost or stolen, you have to manually revoke every single one.