The point is it takes a script kiddy about 5 minutes to scan the whole 4 billion IPs for your port 22 server.
It takes about 90 seconds for the fact that you opened up a random high numbered port that is an SSH service to show up on the list of people that are probably exponentially more intelligent than the normal script kiddy scanning the internet
This does not make you more or less likely to be hacked just for having SSH open. But hey,go go gadget whatever.