Use the PIV applet for SSH and signing Git commits instead? Git supports S/MIME and SSH can use keys over PKCS#11 basically out-of-box on OSs that don't ship gpg-agent (that just interferes with SmartCard usage in general).

This is a per-credential setting, so you can have your SSH signing key be a no-touch key and still use touch confirmation for everything else.

(see "uv" option here https://fidoalliance.org/specs/fido-v2.0-ps-20190130/fido-cl... - the -sk key types in SSH are just a clever way of abusing the FIDO protocol to create a signing primitive)