A few years ago I did stuff like this using Puppeteer. This is not one of the advertised use cases, but normally Puppeteer comes with a fully featured WebRTC implementation, and for my needs, also included the WebAudio spec. Using a combination of these, I could route, transform, mix, broadcast audio without changing or renegotiating the connection details, and also preserve the privacy of the peers by having only the IP of Puppeteer be actually exposed. It worked far better than I had ever expected.
I have been working on a kind of streaming data mixing thing for almost a decade. It's gone through several iterations of using WebRTC and doing other stuff. I am in the process of deploying a new version using some new tech but WebRTC is definitely in the mix.
What you just suggested here is something I had not considered. I am gonna abuse the shit out of this information. Thank you for sharing your experience.