Pretty much everyone I've talked to that uses a YK has the same configuration I do, by priority:

1. Security key

2. TOTP via phone authenticator app

3. Offsite saved backup keys

(4.) Disable SMS

This implies two things: first, that for pretty much every site you rely on, you're going to be both TOTP and U2F anyways, and second, that sites that don't support U2F yet will just start at step 2.

U2F is more convenient than TOTP and, because you're virtually never going to actually use it on a site you U2F into, U2F is still mitigating the phishing risk. You're carrying your phone with you anyways. It's pretty much win-win.