It is not without reason that most common fraud here in Sweden (around 80%) is someone calling under false pretense asking a victim to use the hardware token in order for the fraudsters to create a new electronic id which is then used to empty the bank account, create credit accounts, and so on. So far those security tokens has had a perfect track record from a hardware perspective, as much good that does. Maybe that is a bomb waiting to explode but it is hard to see a stronger incentives for criminals than what already exist.
When people here discuss electronic voting, security of the hardware tokens is not usually brought up. It is rather all the other elements which makes it a bad choice. There would be no physical ballots that can be verified, questionable anonymity, and a loss of a controlled voting environment. All the common solutions to those problems usually results in the conclusion that it then easier to just keep the current system as is.