There's no shared secret between just the bank and you.
Therefore the bank can't establish that it was you, who leaked the secret. (Because it's easy to leak SSN, it could have been any random company you did business with. Or the SocSecAdmin.)
And even if your shared secret was used to withdraw, there's still a chance of the bank leaking it.
So usually the shared secret's hash is stored.
This works pretty well.
Around here banks ask you to give your PIN for identification, or you have to present your ID, and they compare it with what they have on file. (So trust on first use is still important.) Sure, it's probably not hard to fake these plastic cards, but that's a rather serious crime, and then the bank can pull the security cam footage, etc...
Of course, requiring confirmation on an SMS even when you do in person banking might make sense.