That's why I prefer using Yubikeys (using this setup: https://github.com/drduh/YubiKey-Guide) — and this method times out immediately (just press esc when the "insert card" dialog comes up).
Plus you can have multiple keys. Plus you can use them for gpg and ssh. Plus you can back them up. Plus you can print them on paper.