Can't you piggyback off of the processes for social security cards?
I have family who tried to get a replacement SSN card during COVID and the process has been entirely impossible. This a drawback to using physical tokens.
Also, this doesn't eliminate identity theft, the point of issuing is the weak-point for fraud.
1. The office was physically closed for business during COVID.
2. They needed to get documentation which was physically in another country.
3. They could not travel to that country due to travel restrictions.
I consider people in Mexico way less technically sophisticated than people in the US, and adoption has been going well. So I believe implementing something like that would be possible in the US as well.
The issuing/reissuing process has been solved and is working fine, reissuing can even be done online (key-pairs have a longevity limit).
They already serve in that capacity to some extent evidenced by the various government entities who accept an addressed piece of mail as proof of residency. I don't think there'd be enough bipartisan support to make it happen, but I think it'd work great.
Want to open a bank account? Need an SSN. Healthcare: SSN Employment: SSN
Want a STATE driver's license? Need some form of government ID, depending on the state. An SSN card is often an option.
My point is, to adopt a system like you suggest would require reforming ALL of these systems across public (federal, state and local) and private sectors.
It's not just bureaucracy. Doing this at scale across public/private domains is hard.
You don't need a SSN to open a bank account. Health, I believe it depends on the provider, not sure.
A bigger issue is how to handle it being stolen (if the "card" also needs something like a passphrase it can reduce this risk, but it's probably better to make it relatively easy to "cancel" if you lose yours, and then you have to head to the DMV or whatever to procure a replacement).
Certainly better than knowing 10 digits and where you were born and to whom.
If your problem is that someone stole the token, then you're better off with a PKI token.
If your problem is that you have lost your token, then you're better off with a non-PKI memorable token.