That is why fail2ban allows you to configure a whitelisted IP (your office, VPN, etc.) that will never get banned.