Yupp, been through this pain, though gnupg-pkcs11 worked pretty well for me.