So in this case, arent the two factors a) physical possession of desktop/laptop and b) the Yubikey ? How likely is it you'll lose both if you keep your keyring with you?

Your hardware key is in one place, controlled by you. Your password has been leaked all over the internet.

Exactly - It's similar reason to the crypto keys use passphrases.

What you need is a mechanism to detect loss of contact with the human and revoke. One way is to require several hardware tokens to combine their entropy to authenticate. Again, don't make a password be a part of this, use another token.