Also, it doesn't matter. Passwords, secret hashes, ssh keys; anything can be accidentally acquired.
It's a fair point, but in general security has no silver bullets. Build a better lock, and attackers will build a better lockpick (or go through a window instead).
Lack of universal U2F support (a broader login security spec that Yubikeys supports) is Yubikey's weakness, but it only helps the attackers to be a security nihilist.
More and more sites are starting to support U2F. The biggest being Google (which covers Gmail, Google Cloud, Google Docs, etc), but also Github, Facebook, and Dropbox also support it, among others.
The trade-off between usability and security applies here - prior to meltdown/spectre, authentication cookies were happily isolated, so logging in once every 30-days seemed reasonable. Now, cycling login cookies every day or so is a more aggressive policy that is arguably better, but unfortunately, automatically logging users out after a short period hurts site adoption.
Remember: MFA is a two-or-more combination of: something you know (password, pin, etc.), something you have (keycard, token, fob, etc.), something you are (fingerprint, iris scan, voice recognition, etc.), and potentially even somewhere you are (geolocation).