You can use your GPG key as an SSH key[0], and you'll also only need to manage one key - although that doesn't fix whatever issues you might have with GPG, of course.

[0]: https://wiki.archlinux.org/title/GnuPG#SSH_agent