You can use a trezor and https://github.com/romanz/trezor-agent

The cool thing is that the key is derived from the trezor's seed, so you can have several backup devices. And you need pin+passphrase not just the yubikey.