You need to ask for a new one, with its own key pair (and the old one will be revoked).

As for a possible attacker that found your card:

The card has a PIN and locks out after three failed tries. After locking out, you can only unlock it by going to an IRS office where they authenticate you using photo/fingerprint/etc and then using their machine and inserting an unblocking code that only you have. If you don't know the code, you need to ask for a new card.