My take is that there are two layers of friction:
a) people that care about chat encryption and would be willing to change, already did, to Telegram and/or Signal. "I'm not going to install yet another chat app" is a real answer by a friend of mine
b) no one wants to either host their own server, nor pay someone to host it for them. If it wasn't for me and a one of my friends, none of the people I chat with daily would be on Matrix.
And yes, there is the matrix.org server. Out of the ~13 people I chat frequently with, 1 is on matrix.org. "What's the point of changing apps if I'm still going to be using the centralized server" is another answer I've gotten.
I don't know what the solution to this dynamic is other than us, the power users, setting it up and paying for the group of people around us.