I have some more questions about these smart cards (e.g. it sounds like unique public/private key pairs would be generated on each card which might not help with the master key style scenario HSMs are good for). I'm going to be looking into this for sure.
One more thing. I'm assuming FIPS 201-2 Level 3 cards have not been hacked (there were several smartcard hacks a few year ago e.g. http://www.securitytube.net/video/945
But the software running onboard is, well, just software. It's common for applets to want a PIN before performing signing or decryption operations, and to lock out further attempts after N bad PIN entries. Yubikeys were vulnerable for a while to a logical flaw in the applet that let you trick it into signing without the PIN. Presumably applets can have buffer overflows and logical flaws just like everyone else; it's not inconceivable that you could get remote code execution inside the smart card by talking to it the right way. If implemented correctly, you still shouldn't be able to get the key this way, but you can use it. Also, while there exist readers with onboard PIN keypads, most people type their pins into desktop software that you could phish or keystroke log with remote code execution on the client PC, then have remote use of the card as long as it's connected to a compromised machine.
There's also generally a layer (or two) of administrative PIN/PUK codes so organizational administrators can reset forgotten user PINs. This is sort of a problem for the military when people in remote postings lock themselves out of their computers are there are no sufficiently privileged IT guys around. Some organizations can do this remotely, which is (probably) cryptographically secure but vulnerable to social engineering if you can steal a card and convince remote support to give you a PIN reset code. There's also another code that's necessary to replace applets. All of these are presumably vulnerable to being left at their factory defaults, being leaked by a careless administrator, etc. and not easy to rotate if compromised.
HID iClass (smart card-ish proximity physical access control) credentials are supposed to be impossible to clone without a cryptographic key, for example, except the same key is embedded in every HID reader. You can rip one off a wall, JTAG it, get the key, and sign arbitrary cards for any building that hasn't bothered to generate a facility-specific key (and almost no one has).
In a stored value card application, you probably can't get inside the card and flip a bit to give yourself $1,000,000. But there's a much better chance you can figure out the protocol spoken by vending machines and impersonate one. Security through obscurity is common here, and if cryptography is used the private keys in the vending machines may well be sitting in RAM in Windows XP on a subway platform, or crypto primitives may have been chosen incorrectly, or keys may be available by social-engineering the manufacturer.
Smart cards aren't a panacea. As some CIO-focused trade press mentions, most real-world breaches don't involve stealing authentication credentials. Having super-secret SSH keys doesn't help you if I can get a shell through PHP. But it is a hell of a lot easier for an attacker to get your SSH private key from ~/.ssh/id_rsa than from your smart card - at worst the smart card would let him use it.
BTW, you can generate the keypair on a computer (preferably an air-gapped one) and load it onto a smart card. This would let you have backups. But most applications are for authentication; there is no need to have a backup because administrators can simply associate a new card with the account. Server-side HSMs have different properties.'
(I am not a security expert, just taken an interest in this recently.)