Let’s say you have a channel with 500 participants. Is there even any sane way to encrypt that channel?
Discord gets to 25,000 active users before they move to throwing resources at the problem for a hard max of 500,000 total users able to join (but not be active at once, that number is a little fluffy) so it doesn't seem e2ee is really slamming the breaks on scaling vs how far you could normally get.
Signals approach https://signal.org/blog/private-groups/
Whatsapps approach https://scontent.whatsapp.net/v/t39.8562-34/122249142_469857...
Finally if you allow tying and verifying to real world identity and don't require forward secrecy then the encryption side of the problem is no more difficult than PGP.