In real life spaces, we often decide to switch to a different conversation from hearing a snippet or zoning in to what's going on around you when you're bored of the current talk. In this system, I feel like it would be rude to bring yourself over to a group and then leave again when you decide you're not interested in the conversation. Perhaps being able to mouse over a location to here the audio would be good. I don't think there's much expectation of privacy in a space like this anyway.
I think this spacial arrangement would generally make such social dilemmas easier to handle. For instance, I think it would be more awkward to abruptly leave a room during a conversation than to just sidle away. Leaving slowly could implicitly invite others to follow.
The internet has forced us to create a bunch of new social protocols and norms. Why not design our tools to fit with our existing ones?