See e.g. https://freedom.press/training/blog/beyond-signal-phone-numb...
I am not aware of a blog post that fully explains their reasoning, and what exactly the downsides are of "just adding the feature". I can come up with several reasons myself, but it would be nice to read a good summary.
If you really want to use Signal, buy a throwaway SIM or probably use a Google Voice number to verify and login.
Honestly, I personally don't recommend using Signal unless they get rid of that phone number verification / login process. But I also don't blame general users for not using Element (Matrix) either (Which I think is more secure and works without a phone number, but it is unfriendly to use compared to Signal).
There's a good trick that works around the majority of the problems of handing out your cell phone number: People rarely do it and most are not aware of this, but you can use landline or SIP numbers as well for Signal and Whatsapp and probably others, for which neither SIM-swapping nor SS7 attacks apply. Instead of an SMS, you can receive an ordinary phone call, and you only need to receive it once (and can then disconnect your phone [client] again).