Port obscurity is pointless when there's much better ways to secure your host.

You should be running SSH with keys only, never with password authentication. Passwords are for encrypting your keys on your local machine and for sudo which is restricted to specific accounts. Source IP filters really slow you down, but may work in some circumstances.

As your network gets larger you'll need to restrict any inbound SSH and use a VPN to a firewall or bastion host and use something like puppet to distribute your ssh keys. Make sure you do two factor auth to the VPN with a password and at least some sort of key.