Thanks to all, and especially this post, for detailed security recommendations. I hope they convince others that security is important and not to be taken lightly.
I don't really think that security-by-obscurity alone is a good enough solution. I already disable both root logins and passwords on my shell, and have something that tails logs to firewall IPs that are doing nasty things that show up in my logs. It's cute that you assume I'm running Linux though ;-)
I ended up firewalling the entire /14 that /24 was assigned from, because, well, China bothers me...