Ssh ports are brute forceable, passwords have a much much larger search space.
When your SSH port is something like 53148 and you see password brute-force activity in logs it's almost always mean that somebody intentionally scanning your server.
https://www.shodan.io/report/uMZDnWfT
This is a long-running problem and one with various popular solutions: restrict the source networks which you accept traffic for, disable password authentication entirely, and add some sort of rate limiting (e.g. 2004's fail2ban) for failures. Trying to reduce log volume by obscurity is futile - you really need to address the root problem and use tools which allow you to filter and aggregate effectively.
Do you actually have untrusted users on the box?
Why would you not secure the custom port to root-only?