I agree, and I've always thought that the people opposed to it are getting the 'security by obscurity' thing all wrong and taking it to ridiculous levels. Denying information to your attacker is virtually always a good idea. The only exception is widely-used, core stuff like encryption algorithms, web browsers, server framework, etc.

So sure, change the port and all, just as long as you're aware of what it does and doesn't do. I don't think anyone believes that you can allow root login with a password of 12345 if only you change the port, but it can be a good layer in a well-designed system.