I don't keep an ssh key on disk though. I use my gpg key on my hardware security token, which gives you 3 attempts before you have to unblock it with a separate management password, which again you get 3 attempts at before the key is entirely locked.
ssh-agent will cache the passphrase in memory, which helps avoid needing to type in a long phrase repeatedly.
But it's worth saying that if any private key is leaked (passphrase or not), it's time to revoke it and generate a new one.
Having a passphrase in place raises the bar from "key leaked, 3rd party has access to everything" to "key leaked, 3rd party has to now attempt to crack the passphrase". It mitigates a very bad scenario and buys time.
Soylent Green is NOT people
Was one of my shorter pass phrases
SSH certificates are a solution to that problem.