You would also need to enter your username or email address, though, and that's hard to do over the phone... I guess voice-to-text is improving, so that might be an option.
Edit: I guess that may not be sufficient to identify you, but it could verify that you are the account holder for other services.
You really think they are going to be happy about sitting around typing that in on their phone hoping they don't make a mistake ?
With standard numbers, you could use any touch tone phone.