The far less common but much more dangerous attack is a malicious third party intent on gaining access to your servers specifically. Hiding a service on a different port isn't even going to slow that attacker down - they'll use a port scanner to find every port that's listening. The service is going to be found regardless of whether or not you've changed the port. You could certainly mitigate the problem by modifying the service not to output anything until the user is authenticated, and you can use a port knocking strategy to stop it connecting on the first try, but those aren't really 'obscurity' per se.
That's not to say you shouldn't do it if you want to; I'm just not sure it actually makes anything more secure.