Also it's two pretty different things. Signal is mostly personal 1on1 and Matrix is big chat rooms.
That being said I find Matrix usable today and it seems better to bring my friends to that instead of trying to get them to move twice.
The big killer feature of Signal is that once i "convert" someone to use it they always have some people from completely different social bubble that also uses it and that app immediately becomes more useful.
This wouldn't be possible without the phone number. I've used to share your view but nowdays i think Signal way is actually genius deliberate decision and one of the main reasons why it's picking up steam. Encrypted messaging with my buddies has never been an issue I need to replace Facebook Messages. I've realized this when some random Airbnb host started to naturally communicate with me over Signal instead of SMS without even mentioning it.