I thought of a similar issue.

It could be limited to active high karma participants of the community, however anyone can game that with GPT powered bots.

Another possibility is to have yearly oustings instead of voting mods in.

At 90% disapproval mods get kicked out. This along with brigading detection should cover damage by bad actors for the most part.