Maybe I'm misunderstanding your point, but an attacker will also need physical access to your authenticator. That's a much less scalable attack vector.
As for a real problem, others pointed out phishing already. U2F and Webauthn aren't just a certificate management solution, but a protocol for the user agent to make trusted statements to the server.