It is not present in the RAM with smart cards, and especially never with server type HSM:

https://wiki.archlinux.org/title/SSH_keys#Storing_SSH_keys_o...

And even the password can be forced to be re-entered by the agent for every use, if that level of security is wanted.