https://www.funtoo.org/Keychain can do both gpg and ssh

A simple wrapper around pass might do the trick. Someone just needs to implement an agent that does this (tho I'm sure someone out there is working on it).

GPG already allows you to authenticate and your SSH key is protected by GPG.