How do we enumerate which human freedoms that nobody has the right to deny? What if there isn't consensus? If there isn't consensus and they are just imposed by fiat, then you've simply come around to authoritarianism from the other direction.
That's a hard question, like any other question concerning human freedom. I suppose a good rule of thumb is "if you're not forced to subject yourself to it, you don't get a say in it".