Which is more secure, carrying around your ssh private key on a USB or something so that you can connect to your server when you need to, or a long password, say a >15 character alphanumeric? and what happens if you lose your usb?
Require a touch to sign, put a password on the key if your paranoid, if you really paranoid disconnect the yubikey when not in use.
https://www.yubico.com/products/yubikey-bio-series/
(And I'm sure they're not the only company who makes such devices, they're just the one I'm familiar with myself.)
Might be possible on other systems with a fingerprint reader and TPM, https://news.ycombinator.com/item?id=36920105