This is the reason to encrypt ssh private keys with a passphrase. If the key is leaked it's still protected by the password.
It's a built-in feature of ssh. For an existing key downloaded from a cloud provider, use ssh-keygen -p to add/change the passphrase.
Soylent Green is NOT people
Was one of my shorter pass phrases
ssh-agent will cache the passphrase in memory, which helps avoid needing to type in a long phrase repeatedly.
But it's worth saying that if any private key is leaked (passphrase or not), it's time to revoke it and generate a new one.
Having a passphrase in place raises the bar from "key leaked, 3rd party has access to everything" to "key leaked, 3rd party has to now attempt to crack the passphrase". It mitigates a very bad scenario and buys time.
I don't keep an ssh key on disk though. I use my gpg key on my hardware security token, which gives you 3 attempts before you have to unblock it with a separate management password, which again you get 3 attempts at before the key is entirely locked.
SSH certificates are a solution to that problem.