First think of it not from your perspective as a user, but from the site's perspective. If they don't support any form of MFA all they have to authenticate a user with is their password. It's impractical to enforce good password hygiene. If a user's password gets exposed by any means (they write it on a post-it, they get a keylogger installed on their laptop, they are successfully phished, etc), there's really nothing the site can do about it. If they do support MFA and require it on accounts, an attacker then has to get their password like before but also somehow get control of their 2FA device (whether it's a Yubikey or an MFA mobile app, etc). That is much, much harder to do. Not necessarily impossible, but there was a recent article from Google mentioning that they haven't had a single phished account (that they know of), since they mandated hardware MFA for all engineers.

Best practice for a user would be to use a good password manager (so you can use long, unique, secure passwords) and MFA. The second part of that is something that can actually be enforced within an organization.

As far as Yubikey vs software TOTP, etc, it's a bit theoretical. AFAIK, none of the auth apps have had compromises, but it's a lot easier to imagine someone out there figuring out a 0-day attack on a piece of software running on random Android and iOS devices than on hardware like a Yubikey. In theory, the way something like Yubikey works, the actual "secret" involved is stored on the device, and all the computation involving it happens on the device itself, carried out by hard-coded firmware.

As a user, I also really like that the Yubikey (higher end models at least) can store GPG keys and perform those operations securely. So I can set up GPG auth for SSH to servers, and sign my git commits using my Yubikey and know that my private key won't be exposed even if, eg, there's a trojan installed on my workstation. (obviously anything done while working on a trojaned machine is suspect, but the key itself never leaves the hardware, so they can't get that).

U2F literally stands for “Universal 2nd Factor” so it’s not a replacement for passwords (the first factor). They are not comparable.

Not using a U2F key makes you susceptible to phishing attacks.