As for productivity, I don't think the size of the group changes the fundamental problem. For a lot of people, having the possibility of a coworker popping in and disrupting your concentration would be super annoying, even if it is only your immediate team. That's why, in open offices, people often wear big headphones to signal "don't talk to me unless it's important, I'm focusing right now."
That's why I think this would be super useful in cases where I'm willing to sacrifice focus for high-bandwidth communication. For example, during a company hackathon where I'm working with a small team and we're all batting around ideas super quickly. Or during an outage where the team trying to fix things needs to communicate fast. But for day-to-day work, even with my immediate team, I'd much rather rely on asynchronous and written communication that won't interrupt what I'm working on.