End to End encryption in conferences of >2 participants causes substantial quality degradation for the same bandwidth use, since you can't have a central server re encoding streams to produce low quality streams for those participants who need it.
I believe zooms reputation as being more likely to 'just work' in part hinges on that,
That's a fair technical tradeoff, but you can't have it both ways.
If end-to-end symmetrical is not feasible then use fully homomorphic public key encryption, which will allow operations without decryption.
Show me any system allowing video re-encoding with this...
Right, so I suppose that the numerous gateways zoom and others need to offer are an additional problem which implies that these services have to do "something fishy" on the signaling server. Generally, WebRTC traffic does not need to go through a centralized sever, though. It's peer-to-peer after signaling if possible, and if not it can use routing servers that merely route encrypted traffic. So I am wondering if these providers largely make use of standard WebRTC infrastructure plus gateways and how much proprietary magic they have on top/as an alternative. Of course, in no scenario the traffic is really secure.