With the ssh-agent defaults, if I have root/sudo access on a server you are logged into with ssh on, I practically have free use of your private SSH key.
I use gpg's ssh-agent emulation (See the man page for gpg-agent under --enable-ssh-support), and store my ssh private key on my yubikey. With this setup, the yubikey's light flashes when something wants to use my private key, and you can just touch the button on it to approve.