How to secure an Ubuntu 12.04 LTS server

I really think it should not. I run ssh on the standard port, and as long as you use something like denyhosts and public key authentication, and you disable password and root login, you should be good. You could even limit the IP ranges allowed to connect to ssh, but that kind of limits your ability to connect from different places.

The little malicious traffic you experience by having ssh on port 22 is dealt with by denyhosts, the attacker would not be able to get in and they would never be able to try again from the same host.

I moved the front door of my house to the back, haven't had any burglars brake in. Can't argue with that!

The the only reason I change my ssh port is to reduce log file noise. Not as a security mechanism.

Oddly enough, I have also run across networks where outbound port 22 was blocked. Handy for that too.

Me too. I've used a different ssh port for eight years. Not a single log entry from ssh brute forcing idiots during this time.

What's wrong with changing the ssh port?

Security by obscurity. It is trivial to use a port scanner to discover open ports(including your "new" ssh port).