Simply changing the default SSH port cuts it down even more, and I'm not convinced that fail2ban actually adds much security. It may even increase your surface of exposure, if it happens to have a security bug in it.
Now the failed logins on ssh is down to 1-2 per day instead of hundreds.