at least with one of my banks customer support centres this wouldn't happen, if you stumble for a split second they shut down the call and tell you to go into a branch to verify your identity, this is pretty annoying...
(Anyway, I like the idea of using answers to security questions as hard passwords.)