I don't think using different keys per service or host buys you much in terms of security. Yes, an attacker that compromised your keys would have to brute-force your passphrase for N keys rather than one private key, but that just takes N times as long, not really an obstacle.

It would be better to use a hardware authenticator like a Yubikey to either generate a FIDO token-backed SSH key or a GPG key and use the gpg agent as your ssh agent. This way you get SSH keys that cannot reasonably be compromised by other means than physical attacks (someone steals your key and coerces you to reveal pin code).