Though you'd need both keyboard and display built in to verify what exactly it is that you're signing/allowing access to..
1. Send message to sign over the wire
2. Display message to sign on the device's screen
3. Send a message to continue or reject over the wire
4. Device displays scrambled pinpad
5. User enters PIN on the compromised computer, using a blinded pinpad (like the Trezor)
6. The blinded PIN is sent over the wire to the device
7. The device verifies the PIN, and if correct, signs the message displayed in step 2
8. The signed message is sent over the wire to the compromised device
[1] https://doc.satoshilabs.com/trezor-user/enteringyourpin.html
[1] https://github.com/bitcoin/bips/blob/master/bip-0032.mediawi...
But you have the bulk problem... it's a tough industrial design problem.
Tokens are fine to ensure that credentials cannot be easily compromised and to provide 2FA.
PED security is really critical when the goal is to duplicate the token e.g. credit cards if your machine is compromised then any data protected by the Token can also be compromised as soon as you access it.
When in doubt wipe, while it's nice to have a robust security stance in this case I don't think i would matter much.