The most straightforward way to handle adaptive audio/video quality in a multipoint video call is to send the highest quality the sender can to the server, and have the server transcode that to participants in varying qualities.

If it's E2E, the server can't transcode the stream. Realistic options are the sender sending multiple quality streams and the server picking the right one for each receiver, or sending to the server at the quality of the least capable receiver.

There's a concept of bitrate peeling, which would be great for this --- send the highest quality to the server, and the server sends a truncated stream to receivers with poor bandwidth etc. The transport stream would have to be designed so that the truncation points are known to the server, and that the receiver can verify integrity at any chosen truncation point. There's the real problem that this isn't a productionized concept; AFAIK, it's only an expiremental feature in Vorbis, and in that case, quality at a given bitrate is inferior when truncating to that bitrate vs directly encoding at that bitrate.

Without E2EE, the server can directly access and manage how the video and audio streams are sent to each client. It can downgrade video for participants who are struggling, prioritize throughput of audio data for certain clients, prioritize higher res video only for the user who's currently speaking, drop individual frames for clients that are getting behind, etc etc.

With E2EE, by definition, the server becomes a dumb relay, and all control moves into the clients. Depending on how E2EE is implemented, you then lose the ability to do some or all of these optimizations.

There are potential workarounds, but each comes with further tradeoffs in turn, in performance, privacy & complexity. That's not to say that it's completely impossible to optimize given E2EE, as you can move many optimizations to each client, or try to have clients expose just enough data for servers to optimize traffic without being able to read the contents (e.g. by splitting the audio & video streams so the server can manage them independently, or having each client upload separate high & low res video of their own video stream). It definitely makes many optimizations dramatically harder though, creates some serious engineering problems, and in practice rules out some optimizations completely.

Very simplified:

Client sends a video stream to zoom. Zoom reencodes the video on the fly in different formats and forward to the 10 participants separately.

Participants have different devices and network connectivity, that's how the dirty real world is, so zoom has to do that. It's development work and it's compute intensive but it's a hard requirement to have "good" videoconferencing.

Imagine the same thing with E2E. Client sends a video steam to zoom. Zoom can't do shit with it because it's encrypted, it can't be decoded it can't be reencoded.