I think it sometimes gets lost that there's a difference between being harshly critical of another group's opinions, and threatening or harming another group. There are multiple ways in which they're different, and they're not both free speech (IMO).