This reminds me to move to passwordstore with a gpg key on an Yubikey.
Additionally, you can always generate GPG keys on your machine, transfer them to the YubiKey, and then delete the keys from the local machine. It depends if that's an acceptable exposure for your threat model, but for me, having the keys locally for a couple minutes is fine.
That way I don't have deal with different subkeys and other complications that just makes everything hard to understand :)