Perahaps not directly, but it does add to cognitive overhead and makes securing the system as a whole more difficult.

If your environment contains multiple ssh servers running on non-standard ports you would not notice when something out of the ordinary happens.

Something listening to an unexpected port should be something you must look into, not shrug at.