I switched my SSH keys on my Yubikey from using the OpenPGP applet and all the funkiness of gpg-agent over to using X.509 certificates with the PIV applet.

Added bonus is that not only does it work with SSH, but you can use those certificates for TLS mutual authentication in web browsers. Works out of the box on Mac/Windows.