Password based ssh authentication should be used approximately never.
Under Windows, you can forward your smartcard over remote desktop. It's one of the few things Windows has I miss on Linux.
To your point, I wonder whether that consideration holds when the private key is held on an external device, like is the case with a YubiKey. I use that setup, and I can't add the key to the ssh agent.
$ ssh-add .ssh/id_yubikey_gpg.pub
Error loading key ".ssh/id_yubikey_gpg.pub": error in libcrypto
They're specifically refering to password authentication to make the ssh connection.
I prefer to have sudo ask for a password when I'm physically in front of the machine, but not if it's a remote session (e.g. SSH from my laptop to my desktop).
Maybe the SSH agent on the client can re-authenticate to the server when requested?
There is a PAM module that does this: https://github.com/jbeverly/pam_ssh_agent_auth
Note that this is a bad idea from the security standpoint, as it requires SSH agent forwarding. Which means that, if the remote server is compromised, the attacker can use your SSH agent to log into other servers as you.
And your terminal may come with a password manager too, which would be unlocked with whatever means.
Again, on a Mac with iTerm you can do this with a fingerprint.
Fairly trivial to make, at least with NixOS.
Otherwise I think you just prepare a certificate ahead of time, and scp it during the first connection, then immediately disable password-based access, or at least change the password. Any passive eavesdropping still needs to defeat the encryption somehow (no feasible ways are known now), even having seen the initial exchange.
If you have an active MITM attack, all bets are off, because the attacker could even grab the image with the pre-baked key you're sending, and copy or change the key. If this is not possible, then the pre-baked key would help. If your security is really important, don't use ther cheap GoDaddy's offerings with limited SSH.
[0] For example, you could let the user write their key to an SD card and then mount it ro on the device.
Is that uniform across all devices, or device-specific?
Practice I've seen for some years now is to have a label on the device with admin/root password, which is presumably neither uniform across devices nor trivially-determinable from device characteristics (e.g., MAC address, sequential serial numbers, etc.).
I'd still consider that practice reasonably tolerable, though you should be keeping better tabs on assets and credentials.
There is a role for a common secret in a secure ecosystem (password, passkey)
For bootstrapping you can bake a bootstrapping key into your installer which is removed after the system is configured.