This is why I come to HN - brilliant ideas that I should have thought of before, but didn't!
I just changed my server, and uninstalled the - now truly useless - fail2ban. I use SSH keys of course, but without fail2ban my server's logs were constantly flooded with hacking attempts.
No longer - wireguard for the win. Thank you, chlorion!