To log in via SSH, you (should) have a key, and a password for that key. To sudo on the server should require a different, complex password for the paranoid admin. (So 2 long passwords and a key to gain sudo priv's)