It is possible to use brute-force via timing attacks to determine whether accounts exists on hosts, and then either focus different attacks on those accounts or spam them.

It is amazing how much this needs to be repeated, but security is not an absolute. Fail2ban adds another layer of security, thus it is a good thing.