That actually sounds pretty ideal. Add 3 people to your killfile, and you've removed most of the noise!
Thus nobody new joined the groups and they stagnated.
I quit Usenet 20 years ago and if I look at them now some of those same trolls are still around. Or they have migrated to facebook/twitter.
“Moderation” can be implemented by a cohort of likeminded people by sharing and maintaining a kill file, similar to how mods on a subreddit determine who to ban. But they only are effectively “banned” for the people to trust that cohort and agree with their decisions enough to “subscribe” to their kill file. Don’t like that they banned too many people you want to hear from? Fork their kill file, or subscribe to a different cohort’s kill file.
Make the UX good enough and it may be possible to satisfy the “free speech” absolutists as well as the people who want a well-moderated platform without trolls.
But like I said the problem was the new people. They'd open up the group see 90% junk and not bother going any further.
This is extremely demoralizing.