Nobody wants to use a messenger that's public, or that with a single centralised compromise/hack could make all their messages become so.
To me that disqualifies them as even being potential solutions to the problem.
Matrix has significant usability issues in its currently-available form(s) - but it at least meets the most basic of requirements that user messages are private, and does so fundamentally.
Really though, why is end-to-end encryption the one thing that should disqualify a system? What about a10y, i18n, etc.? For a vision-impaired user a10y matters a whole lot more than encryption. That is the real point of openness and federation -- allowing users' needs to be met by any developers who have the time and inclination to do so, rather than leaving users at the mercy of a single team (or even a single person).
Because security as an afterthought is known to fail.
That is what being religious about these things gets you. Signal is completely closed and its users are at the mercy of the developers. XMPP+OTR was a kludge but it worked and the only reason the outstanding problems were not resolved is that the cryptographers behind OTR decided to focus on Signal. Matrix is basically irrelevant right now and whatever security benefits it might have are equally irrelevant because very few people are using it.
Telegram's far better UX than Whatsapp is possible because they do not have e2ee on anything but Secret Chats (and I've never seen anyone on Telegram using them). See, for most users the promise of security is better than actual security and all attached UX problems.