it's not supposed to be a secret in the "something you know" way, but rather "something you have" - i.e. the physical card. If they store it you no longer need the physical card for an entire family of attacks & frauds.