Same can happen to you.
Same can happen to you.
The next level of mitigating that sort of a thing is to have keys not be on the hosts at all. Enter HSM - Hardware Security Module. A wildly complex topic I cannot hope to cover in an HN comment, but fundamentally, the private keys are not on the same HW as the server software which needs them.
A fundamental property of an HSM is that you, the HSM user, don't actually see the private key. You can ask the HSM to generate one. Derive things from it (in the cryptographic sense of derive). Even prove the provenance of such derived data. But the HSM should not reveal the actual private key.
In the cloud world, the HSM equivalent is known as KMS (Key Management Service), and the Good cloud providers all let you manage your own (with the downside being that you now need to manage your own keys).
Not just. The disk itself may be encrypted, the keys may be encrypted. Doing this unnoticed and without rebooting would require modifications to the hardware.