A native server with a socket-based and not longpolling based transport system would indeed solve many of the issues.
But even then, Matrix will likely not scale well enough to replace Twitch's backend, or to be able to nicely handle QuakeNet's event channels.
You'll also have to have a native client at that point, because the webclient also breaks down at those amounts. And you'll need to use a binary protocol or simplistic text protocol between client and server, because otherwise the user's mobile connection won't be able to handle that traffic.
Matrix will have to change a lot of things before that scaling is easy.