Yeah. Actually ssh agent speaks PKCS#11 (both client and server) so it's possible to interface with the hardware token quite easily. I'm using that to store my client key in TPM for example.