You can. But, if I lose my SSN card I can still (usually) use my SSN because it exists in a functional form inside of my head. If it was a PKI card, I would physically need to have the card.
I have family who tried to get a replacement SSN card during COVID and the process has been entirely impossible. This a drawback to using physical tokens.
Also, this doesn't eliminate identity theft, the point of issuing is the weak-point for fraud.