>there is no good way to get around this

An authoritarian can minimize the influence of the extremists or other people that vote in a way contrary to the well-being of the community.