The problem is that you are also at the mercy of passwords selected by your users for smtp, imap, etc. So you still need some defence against brute force.
For administrative protocols (ssh, rdp, etc), I am a firm believer in IP whitelists, which give you the additional peace of mine of protecting you against future zero days, unless they affect the firewall.