The problem stems from the fundamental fact that a very small percentage of people are assholes, a faction of people are intent on forcibly silencing and harming assholes for being assholes, and a faction of people are intent on not permitting that second group of people to do so.

That's the problem: two groups of people are prepared to use force to enact their contrary visions, while a third (relatively small group) is catalyzing the conflict.