I can't emphasize this one enough. Unless you need to login from a lot of different machines, there really isn't any excuse not to do this. It also has the bonus of making logins really easy since you don't have to type a password.

Also, yes, among with changing the port the only way that should be possible to get in is through 'keys.