> If your keys are not protected by a passphrase

Looks like the private key can be leaked regardless of the use of a passphrase, but you'd get the encrypted form that would need to be cracked offline.