passwd -d destroys the ability to a) log in from a console and b) to use sudo, which might well be a dealbreaker for most people. Disallowing password auth in your sshd_config is typically a better option.