> More concretely, that means if the person sitting next to me at work uses a smartcard granting them access to private documents, and they leave it on their desk during lunch, I can clone the smartcard and give myself access to all their sensitive files.

I'm very skeptical that this is actually possible with proper Javacards. In any case you shouldn't leave your smartcard lying around. And furthermore what kind of system uses RSA for document/disk encryption?