Does it _harm_ security?
If your environment contains multiple ssh servers running on non-standard ports you would not notice when something out of the ordinary happens.
Something listening to an unexpected port should be something you must look into, not shrug at.
Moving to non-standard means more sophisticated discovery and possibly a more sophisticated attempt.