A rouge employee could steal a physical key left lying out, much like they can steal keys from computers.
You only get some limited number of pin attempts before it locks you out.
A stolen key is useless for gpg/ssh.
No one in that transaction cares if it was lost or stolen.