Tying 2FA to hardware is for most of the common use cases a bad idea. Instead always use TOTP and keep the seed in a secure storage with multiple backups.
If on top of that you like to keep it on your phone to generate the code that way, fine. But at that point you can destroy the phone and it doesn't matter, you'll still have access.
> While I don't like passwords and understand their very real security limitations.
When used correctly, password are a fairly great solution with fewer limitations than competing solutions. By properly I mean generate yourself 128 bits from /dev/random, never reuse and store securely.