What you're running into is almost certainly noise cancelling. It's turning off your incoming audio so it doesn't intrude on your microphone picking up your voice.
Disabling or tweaking noise cancelling or audio modes (i.e. to "headset") on basically every meeting software will eventually get you a combination that doesn't do this. It's sometimes a bit hidden though, and many have chosen to just say "everyone gets maximum noise cancelling" rather than trying to guess based on your audio device(s) so it doesn't always do it automatically or obviously.
Not triplex or quadruplex etc.
In other words, if you're having a 1-1 video call, both of you have your audio working at all times.
But as you go to 3, 4, 5, 10, 20 participants, it generally continues to be just max 2 simultaneous audio streams, determined by whoever has been the loudest recently.
Unless you turn on special features like music mode etc.
This is a feature, not a bug, because otherwise background noise and sounds would start adding up to become intolerable. (Why we usually try to intentionally stay on mute anyways, so we don't accidentally become even just that second audio stream.)
I was talking to someone who specialises in corporate communication at a conference last week and she confirmed my feeling above as well. Her thing is “talking over each other is a key part of human interaction and forcing one at a time is stifling and unnatural”
You could reduce sample size, and send fewer samples per packet to reduce total delay, but overhead goes way up (overhead is near 50% at 20ms samples, one per packet). In theory, you should be able to do something nice for people doing audio and video by including audio on the video packets, but it's not simple, so I think most conferences don't do it.