Let's start with SSH. It's just plain unstable. If you try to set this up, you WILL have constant stability problems. I can assure you I have both personally experienced them and asked several other people who have attempted it to confirm that they too have various timeouts or other errors.
One alternative is PIV. PIV is even more of a pain in the ass to get set up, and has similar stability problems.
If I have a prescriptive recommendation for SSH, set up Duo for 2FA and use Yubikey-AES in "long press" mode (putting the key in slot 2). This gives you a strong hardware-backed authentication factor and protection against "Yubispam" (i.e. accidental credential leakage, a problem more severe than most would consider it)
As for GPG, well... GPG is a tire fire. I also went through the Yubikey GPG PIN bypass (not their fault, but since Yubikeys cannot be field patched, this involved physically rotating every key, which Yubico replaced for free, but still). GPG suffers similar random faults, and gpg-agent can die in a fire, but at least unlike SSH you're probably not using it as frequently. Still, getting things set up is ridiculously arcane, thanks to gpg being a byzantine tool and the crazy mutable state nature of ~/.gnupg
If you end up attempting to replicate this sort of setup, I can bet you dollars to donuts you will wind up asking yourself "Why did I think this would be a good idea?" after a few months