The problem is the social graph. You need to know the ID of all your contacts at some point, right? Either that's stored on the server (e.g. your Facebook friends), but then the server has access to it, or it's stored locally in your app, but if you lose your phone then you lose your graph. Signal has been working on a way to store your graph securely on their servers, but it's hard, and not there yet.

The great point about your contact list is that it is decentralised, and it exists already (so you don't have to exchange an ID with all your friends manually).