When last I checked, at least on linux/ubuntu the blocking issue was in opensc-pkcs11, not ssh.