Non-technical people I've introduced to Signal on iOS stop at the "upload your contacts" stage and go "I don't want this random little company to have access to my contacts". These are non-technical people who understand the importance of keeping their contact list private, and there are a lot of them.
Making it a de facto requirement that users on your network be personally identified via a phone number ("get a burner number"? really?) and that you have a copy of their contact list and phone calls, all within US jurisdiction, means mass metadata surveillance a single judicial order away (as it is now with the telecoms).
Unlike POTS, Signal prevents phone taps, yes, but it does not hide your social network, and so does little to provide you with the ability to associate with someone anonymously.