Enable two factor authentication on your servers, or else two factor on a bastion server and disable external ssh other than from your bastions on all other servers.

I strongly recommend YubiKey, it is convenient and cheap and extremely secure.

With ssh it is very easy for someone to create an ssh key that does not have a passphrase. With that, it is possible to log into the server with just the key file and nothing else. At that point all it takes is someone to lose a laptop or leave a computer unlocked and unattended and someone can get access to your machines.