This assertion confuses me.
I use fail2ban on boxes I have key-only ssh configured for.
Are you aware fail2ban works for services other than ssh?
If an attacker / script knocks unsuccessfully on my ssh door, other doors are then closed to them.
I also get much (much!) cleaner logs thanks to fail2ban.
I suspect that you're confusing fail2ban and port-knocking (or using fail2ban as a port-knocker).
The point of fail2ban is to prevent an attacker from brute-forcing your server. In a key-only config, the chances of getting brute forced is smaller (by a few orders of magnitude) than getting hit by an asteroid and having the server get hit by an asteroid, so fail2ban doesn't really help.
_In theory_, the same would be true for port-knocking.
However, in practice, sshd can have security holes which a malicious scanner could exploit. And while port-knocking doesn't help against a determined attacker (it's subject to MITM, replay-attacks), it does help with defense-in-depth.