Not your contacts, but everyone [already using Signal] who has you as a contact.

To remove that functionality would be equivalent to removing the ability to look up if a Signal message can be sent to a particular number. At least as long as phone numbers are used as the sole identifier for Signal accounts—another decision that has been debated into the ground, of course, but was done for understandable reasons.