Funnily enough, changing the ssh port to a random unprivileged one is now possibly the easiest and most effective step to harden a box... I guess it shows how the internet has changed.
In the last two months, I've been getting lots of bruteforce scan attempts on all my boxes that run ssh on high ports, so it seems this is no longer as effective as it was :-/
You are not alone in this. This week I actually changed ports for ssh because of the sheer number of brute force attempts. We'll see how long that holds out.
Why not just restrict access to specific IP blocks? Even if you left it open to Verizon's entire IP space so you can hit it with your cell, you would still dramatically lower your incident rate.
Yeah I could do that I suppose. I host several people on it however. Right now the unique port is enough.
Why not just use fail2ban or similar, to automatically block the ninnies for a while after a few attempts?
Yup, using LFD for that purpose. There's still a lot of attempts.