Could you please expand on your point ? I understand that security by obscurity is not optimal, but I don't see a good reason not to switch ports.
[1] https://en.wikipedia.org/wiki/Principle_of_least_astonishmen...
Changing SSH to a non-standard port is an inelegant solution and doesn't actually bolster security.
If your environment contains multiple ssh servers running on non-standard ports you would not notice when something out of the ordinary happens.
Something listening to an unexpected port should be something you must look into, not shrug at.
Moving to non-standard means more sophisticated discovery and possibly a more sophisticated attempt.