Why?
The benefits are largely theoretical if you choose sufficiently strong randomly generated passphrases, with some symbols and numbers mixed in, which I set my password manager to do. (Actually, I have a couple of super important, super long ones I keep only in my head.)
The draw-backs however are several:
1. I find that its not uncommon I find myself having to chain together ssh tunnels and other strange things to debug networking issues and need to login to another machine from a new machine that doesn't have the ssh-key. Treating servers as cattle exacerbates this issue since it's more likely to occur.
2. If a machine or server I use to manage stuff as a gateway/proxy/vpn entry to my network has all my ssh keys on it, and it becomes compromised because of some 0-day, the attacker now basically has access to... multiple entire networks. Now, this is technically true of my password manager as well, except that the context / IP addresses, etc, is not going to be as obvious as it will be on the server where the attacker can see the running services, check logs, command histories, etc. With the password-based management, sure I could get hit by a keylogger, but everything won't be compromised by default.
3. In my experience, you're more likely to lock yourself out than let an attacker in. You basically still have to have your keys managed via some type of password manager anyway because otherwise you run the risk of getting locked out of everything in case you're away from your primary machine and need to address some emergency or something.
Public key authentication isn’t just providing more entropy than passwords.
Passwords (as used in SSH) are bearer tokens – send yours to the wrong server, once, and you‘re compromised, for this and future sessions. That’s not the case with public key authentication.
> gateway/proxy/vpn entry to my network has all my ssh keys on it, and it becomes compromised because of some 0-day, the attacker now basically has access to... multiple entire networks
That's why you use SSH agent forwarding https://docs.github.com/en/authentication/connecting-to-gith..., so you never need to copy the private keys to other computers, and thus they can't be stolen from there.
Your private keys shouldn't even be accessible to you, they should be on a secure enclave like a yubikey, and you should forward the token along the chains. No risks, and basically painless, especially if you switch to certs so you don't even have to know the public keys ahead of time on the servers, just all trust the same private PKI.
That’s not the only reason or way to use SSH keys. Even when stored on disk (in plaintext or encrypted) they offer advantages over password authentication, e.g. making it impossible for an MITM to steal credentials or impersonate an authenticated client even without validating host keys.
A man in the middle attack where you accept the server key can steal your password, but it can’t your key.