It sounds like you are trying to justify an anti-majoritarian stance with majoritarian consensus. If you are against majoritarianism, you should probably find another way to make your case. Here, let me help you out:

It's better for a small minority of people to make the rules. A small minority of people agree with this, thus it is correct.