Unless I'm missing something, that essentially solves the issue, no? Persistent private key, usable across all your machines that can't be exfiltrated. Bonus since you can even generate the key directly on the device!
This gets even easier with newer OpenSSH versions if you use FIDO2 auth for SSH. I've not played with it personally, but word is you just plug in the key and off you go