Or someone can simply own your key holding device as you sip your beverage at the local hotspot or home wifi. Simpler to go after the secret privileged client than the server.