Intolerance, you draw the line at Intolerance. If BLM, PETA, Antifa want people of other race/class/sexual orientation harmed in any kind of way, sure go ahead and ban them. If on the other hand they want fair treatment, equality or reprieve from targeted brutality, listen to them and see how we as a society can address their concern. If they resort to violence, arrest them and treat them like you would treat a criminal.
It is not really that hard. Right to free speech does not really mean right to say anything anywhere without any consequence.
The irony in this whole thing is, the more tolerant you are to an inherently intolerant ideology like white-spremacy, the less tolerant we become as a society.