If it is apple you don't trust in this regard, keychain is the least of your problems.
If it is outside attackers. At least they should need physical access to the machine to crack it, right? (If you don't back up that specific keychain to iCloud)
And when I'm spending a month or two in Asia, for example, I have no illusions that I could possibly get another mailed to me, if it broke or was lost.
These are great in terms of security, and should be an option for people who need it, but shouldn't be obligatory (I wouldn't use this for my bank if I had a better option).
The key stays on the card. The card will do signatures and key generation, and also holds login/etc credentials. It works basically exactly like Malka says: bytes go in, signature comes out. It can likely do encryption as well, but you'd probably use it to generate a temporary key and then to sign the encrypted results, since the processor on cards is relatively weak.
0: http://www.acs.com.hk/en/products/17/acos5-64-cryptographic-...
Also, with that old and worn proverb about eggs and baskets, if having none of your eggs is equivalent in value to you for having only some but not all (i.e. you must have exactly all of them, or else you fail), then putting them all in one basket is better than putting in two (or more).
If you're trying to minimize baskets, though, then having two very-secure baskets is much better than having N baskets ever-growing because you can't trust in their sturdiness.
Which is to say, putting a USB with your private key on it in your safe-deposit box at your bank, for example, means being more assured about the fact that you'll have a backup copy, meaning you then feel safe excluding it from local hard-disk backups and synchronized remote backups/cloud storage, and have no reason to have it on any computers you aren't currently using for the sake of having somewhere to import it from. In a sense, you've added one vector of attack, but in practice, you've removed several.