And relying on what looks like secret data (changing the SSH port) where the number of bits of entropy is low enough that it's plausible to try them all (16) probably still counts as security by obscurity -- it might hide you from many attackers, but it's not enough to make you secure.
Relying on data that's not actually secret, just hard to find, is just insecure.
That's not why you change the ssh port at all.
You change the ssh port to filter out false positives, if someone is attacking you on your weirdo ssh port, it's likely an actual attack that you need to pay attention to. You still need to do the rest of the security stuff.
When your SSH port is something like 53148 and you see password brute-force activity in logs it's almost always mean that somebody intentionally scanning your server.
https://www.shodan.io/report/uMZDnWfT
This is a long-running problem and one with various popular solutions: restrict the source networks which you accept traffic for, disable password authentication entirely, and add some sort of rate limiting (e.g. 2004's fail2ban) for failures. Trying to reduce log volume by obscurity is futile - you really need to address the root problem and use tools which allow you to filter and aggregate effectively.
Do you actually have untrusted users on the box?
Why would you not secure the custom port to root-only?