Might be troublesome if you don't want the other person to know it's you, though.

But that's probably not a problem in day to day communication with family and friends.

There are better ways to verify someone is who you think they are, e.g. asking them about an inside joke that you only told them in person at some point in the past, or what art you have up on the wall in your dining room.

Also, you might in some cases want to share knowledge or evidence without disclosing your identity, and phone numbers miserably fail for that use case.