That's an interesting threat model that I don't hear very often. Can you expand on what threats you're trying to protect against? If you ever sign in to Twitter on your phone, you're trusting your phone with your Twitter account at least to some degree.
The main thing security keys protect against that TOTP doesn't is phishing.
And even if you are already signed in to everything and think you won't ever need to sign in on your phone again, the malware can force a logout of something, and you won't be able to know if it was just some software update or expiration that signed you out. So you'll sign in again, and the malware will intercept your security key.
I almost never use Twitter on my phone, and if I switched Twitter to depend on my Yubikey, I would stop using Twitter on my phone completely.
Is remote desktop compromise less likely than remote phone compromise? Phones tend to sandbox each app, so a single malicious or compromised app cannot take over your phone. Desktops on the other hand allow any app to do essentially anything on your computer. Accidentally open a malicious email attachment and you're done for. Generally I consider machine/phone compromise outside of my threat model.
Is security key theft less likely than phone theft? Also, phones generally require a password or fingerprint or face to unlock, while security keys generally do not (Yubico announced a fingerprint-protected one 18 days ago, but it's not available yet).
I bring my phone with me in lots of circumstances where I don't bring my security key, so yeah I think it's more likely to be lost/stolen.
Also, TOTP has a MITM problem that U2F/Webauthn doesn't have.