I'm quite familiar with cryptography and the principles of secret keeping. I am not as knowledgable about smartcards I admit.

Is there something that makes a smartcard based USB key work differently when held by a person other than its rightful owner?

My impression of them, perhaps incorrect, is that they are akin to Hanko ( http://en.wikipedia.org/wiki/Hanko_(stamp)#Japanese_usage ). Basically a system of using possession combined with a fairly unique set of data as a means of distinguishing and authenticating its owner. But in practice its far less than perfect.