Using a yubikey says, specifically, that if I lose this little device and the bypass codes, that I have presumably stored on encrypted storage in a way that doesn't require the yubikey to access, then I want it to either be impossible or exceedingly difficult to recover access to this account.
Very few people actually want that, and if yubikeys become widespread, there will be a wave of people having tantrums because their yubikey is lost and the account is unrecoverable.
If it isn't extremely difficult to recover an account in the absence of a yubikey and the loss of the bypass codes generated on enrollment, then there's no point to them.
I've run a b2c website. There is a shocking percent of internet users -- I'd estimate 20% -- that cannot reliably tell you their email address (5% that literally can't, and another 15 that can't reliably). Those users having yubikeys would be an utter disaster.