Yeah, but say I am an attacker doing some kind of brute force password hack, and I have a certain number of successes.

Given the funnel there, it might well be worth it for me to put some energy into figuring out who the person at the other end of that account is. Phone numbers aren't secrets.