However, proper ingress filtering or local iptables/pf rules would stop any unwanted inbound traffic from reaching your server, and you should definitely be using ingress and egress filtering on your network.
It is basically zero inconvenience to add an extra argument or shell setting.
If you really want to hide SSH from script kiddies then you're better off setting up firewall rules to whitelist trusted IPs. Heck, even port knocking[1] would be a better option than changing the port number.
FWIW, I've never seen a brute force attack on SSH other than on port 22. Most likely, a targeted attacker would realise if you change the port, you're probably not going to have a trivial password.
Random IPs are unlikely to be port scanned but any reasonably popular site will.