With regard to the output productivity of the conversation, there's also something to be said about the "narrative control" that limiting speaking to a single person at a time brings.
With voice platforms (including things like Discord, Zoom, etc.), there's always one thread of conversation going on. The meeting is generally a point A to point B discourse where circling back to examine a topic further is awkward and destroys the overall flow. People want to avoid that, so anyone who can manage to dominate the conversation and stop other people from interjecting has the power to effectively manage what comes next. There's a lot of potentially valuable conversations that get lost along the way as a result.
As an aside, I've seen this bleeding into real-world conversations over the past few years as well. Pre- and post-COVID conversations with my peers in college were staunchly different in how they handled side conversations. Perhaps it's just a product of my cohort getting older and more accustomed to the norms of professional communication, but I've noticed far fewer productive side conversations in groups of roughly 7 or more and a much stronger focus on "staying on-topic".
Text-based platforms have long since managed this issue with reply-based systems, threads, and hierarchical, nested conversations (e.g. HN, Reddit, etc.). This is really only possible asynchronously, though. In a voice format, the best you can do is leave to a breakout room or other channel and come back when you're done, but in doing so you're missing out on the conversation in the main thread. Unless we can somehow find a way to work past that limitation, I'm convinced that practically any platform that relies entirely on voice is going to fail to have the reach and community-building effect that something like Discord or Reddit has achieved.
Additionally, all of this also fails to consider the fact that synchronous-only voice is yet another social distractor that demands your attention on someone's terms other than your own. As seems to be the sentiment on HN, I'm pretty staunchly opposed to normalizing yet another consumer of my time as necessary for social acceptance, but perhaps I'm a bad sample of the public opinion.
Apologies if I repeated a bit of what you said here, but the point I'm trying to make is that you're entirely right about the obviousness of this issue. It doesn't take much consideration of the concept to realize that there's an immediate growth problem. Novel as the idea behind social audio might have been, it was doomed from the figurative (and literal) word one.