Oh yes, the first thing I do is change the SSH port, eleminates 99,99% of the auth attacks.
I can't emphasize this one enough. Unless you need to login from a lot of different machines, there really isn't any excuse not to do this. It also has the bonus of making logins really easy since you don't have to type a password.
Also, yes, among with changing the port the only way that should be possible to get in is through 'keys.