That's an excellent point. I painted myself into a corner here because I am letting some of my friends use the server as well. Have to gently transition them to public key usage only now.
This isn't something they should have trouble understanding/accepting.