>Signal is OSS and you can start your own fork & network if you want to.
Resources and network effects make this a much much less likely endeavor than in federated systems. I can reasonably (and do!) self-host Matrix and XMPP servers for myself and a few friends; I cannot reasonably host a Signal server for everyone I might want to possibly contact via Signal. (And that's setting aside the effort involved in convincing everyone I might possibly want to contact to use my Signal server. It's just not going to happen.)
EDIT: Oh, and I don't think the VoIP side of the Signal server is FOSS? At least I believe that used to be the case.
>App publishing platforms not having a good binary signature verification system is the orthogonal issue that you're bringing up, that would in many ways apply to matrix for most users too. Most will never bother to sideload it.
If the main client developers are pressured to put a backdoor into their client, I would expect non-backdoored forks to rapidly pop up in the stores.
If the platform provider is also legally pressured to not allow any non-backdoored clients into the store... well, I hope that never happens, but if we get to that point I hope more people would bother to learn more and start sideloading things. Maybe wishful thinking on my part.