The walled gardens are using polished UIs and free hosting to win over open alternatives, and both of these costs money. From users' perspective, it's free and convenient.
To break these, we need technology and infrastructure that enables distributed decentralised hosting, and that decouples UIs from servers. But UI developers still need to be rewarded, and hosting needs to be profitable, otherwise they won't be competitive.
It would involve some form of micropayments system, cryptographic identities, and so on. If these things had been in place 20 years ago, IRC would be what whatsapp is today.
The matrix project sounds promising, but I'm guessing it underestimates what is actually needed to replace walled gardens.