I am bridging WhatsApp to my Matrix server. On Matrix it shows the username when the number is not in your contact list, and only then did I notice how unnecessary access to the contact list actually is (as long as you don't start new chats with people)