The problem is not the passwordless sudo but running untrusted programs on your computer under your user. They don’t need sudo to steal your SSH keys or inject malicious code in your .bashrc.