Bingo. You hit the nail on the head. To efficiently do large conferences, you need to both multiplex audio and video at the media server layer and have a concept of active speakers.
"Jitsi Videobridge does not mix the video channels into a composite video stream, but only relays the received video channels to all call participants."