They just need to scan once to find the port speaking SSH, then brute force as usual.
If you want actual more protection run a bogus SSH (with no login allowed) on 22 and a thousand unused ports... then they have to try them all or guess which port is actually able to log in.