My Yubikey requires physical touch to authorize an SSH connection so it doesn't matter if I forward my agent to malicious servers as their attempts to hijack my socket will result in timeouts from my not authorizing their use with my physical key
https://github.com/maxgoedjen/secretive
Extra nice with the new Apple Magic Keyboard with Touch ID.
Modern SoCs do have secure enclaves, but how can one use it? I don't want a Yubikey dongle which i will forget at home when going to the office and vice versa. I'd be happy to get just the secure key storage without the fingerprint feature.