The phone number is not even the worst part really. Signal's privacy policy says they don't collect and store sensitive info, but how is a list of everyone you've been in contact with not sensitive?
Sadly I've had very little luck getting my family to move to Signal and less luck getting them to use Jami. The bulk of my communications are unsecured SMS/MMS. It's such a basic function that I'm surprised open source solutions aren't better. Whatever they've become Signal had a pretty great interface.