Depends on setup, but frequently private keys are inaccessible to the web server worker process. (Which starts as root, loads keys, drops privs, etc.)
If you don't trust it your automation, you rotate the keys manually, as you would normally.
There are no valid reasons to throw the baby away with the bathwater.