As I understand it, the challenge is to do it in a privacy-friendly way, since your contact list of phone numbers is on your phone, but this has to live on Signal's servers.
Signal uses phone numbers because it makes discovery easy. Threema, for example, can use phone numbers for discovery but does not require it. Discovery without phone numbers is easy. I see my contacts and scan their Threema QR codes. If I need to contact a friend of a friend, my friend gives me the FoaF's Threema ID.
Signal would need to store a second contact list if it was not using the phone contacts. And suddenly you need to backup this second contact list. If every app does that you can forget about the user backing up everything, they simply won't do it and the feature becomes useless. The solution would be for Signal to store it on their server, obviously encrypted. But then you have different privacy issues to take care of: how can you retrieve a user's contact without storing its identity. How do you hide the number of contact they have...
every chat application that i have stores its own contact list. in fact i don't even have any contacts in my general phone contact list, because i don't call or send sms to people. and i don't want any chat contacts in my phone contact list.
i have not tried signal yet, mainly because it is not available on f-droid. but if signal insists on storing its contacts in my general phone list then i won't be able to use it. and that's ignoring the problem with using phonenumbers.
there is no technical problem to store contacts locally. deltachat does that too. deltachat also provides a backup feature to export the local data including contacts and messages so you can restore them on another device. there is no reason, signal couldn't do the same.
i don't know why this is so unusual. we are having this same argument every time signal's use of phone contacts is brought up. and every time the same claims are being made.
Every other app can see it if you click "Accept" on the per-application consent dialog ...
it's basically saying: i am going to take over your contact list, and if you don't want those contacts to be shared with other apps, then you can just block them.
what if another app wants to do the same?