The Contacts permission is completely optional, and contact information is never stored: https://signal.org/blog/private-contact-discovery/
Traditionally, in Signal that process has looked like:
The client calculates the truncated SHA256 hash of each phone number in the device’s address book.
The client transmits those truncated hashes to the service.
The service does a lookup from a set of hashed registered users.
The service returns the intersection of registered users.