This is a tricky problem, and we have a couple of ideas. But currently if we're having a longer chat (double clicked will latch-open the mic) then we mute the other groups. Anyone else in the other groups will be able to see that you're muted.

We're contemplate auto-muting other groups when you have incoming audio. Would love to hear your thoughts as well.