Everything, that is, until SSH 8.2 came out. Using a Yubikey (or any other U2F-compatible key, which is a lot of them) is a breeze: Run `ssh-keygen -t ecdsa-sk -f ~/.ssh/id_ecdsa_sk` to generate a key from your Yubikey and you're done. You can even use Resident Keys mode (if your key supports it) to avoid having to carry the private-key-half around with you, you can load it straight from the Yubikey with `ssh-add -k`.
This is the only way that lets you walk up to a machine, plug your key in and SSH to your server securely with just two commands. The downside is that both sides have to be running SSH 8.2+.