The phone number is not required as per the Signal protocol -- it's an implementation detail -- another token could be used. At this time, phone number verification is used in the initial authentication flow to protect against someone else spoofing your phone number and pretending to be you.

See https://github.com/WhisperSystems/Signal-Android/issues/1181