Don't authenticate to servers with a password. Ever. Use public key authentication.

What's wrong with good old fashioned pam modules to put users in sudo/non-sudo groups?