Signal forcing you to give your phone number to anyone you want to contact is a much bigger security problem to me, especially when in my country it's both somewhat hard to acquire an anonymous sim card and a relatively small sum of money can get you a lot of private information from just having someone's phone number.