You can password protect the Yubikey. That's how mine is setup. Adds an extra step, but I don't worry about losing it.