Keys are not the only real security... real security includes something you are, something you have, and something you know.

Having something like this combined with the something you know (passphrase) would be real closer security. Now anyone with your token can access everything that key gets access to, without you needing to be there, or sharing a passphrase. That's less secure imho.