Both of the solutions you have still disrupt the game, even if they do temporarily solve the issue.
It's also like, sort of kind of not how social games work in many cases. In order to make friends in the first place, you gotta get out there.
No, I do not want to be friends with those people. But I don’t know that yet, because we have not met. The only way to meet them is to... meet them.
I really have a hard time imagining a scenario where I can trust the game server admin, but not the admin hosting my voice server. Especially if it's some team-based game.
Also, not every game that uses Discord has the “server admin” model. Most of the games I play do not. Still use Discord though.
Here's a convenient, albeit somewhat outdated pastebin explaining how https://pastebin.com/ViWmrsDJ