I was frustrated with the complexity and automated the process. Unfortunately even automation doesn't protect you from the complexities of GPG, but it does help. This is the tooling: https://github.com/ixydo/gpg-smartcard-automation