We have to come at this very directly from the perspective of the outcomes we wish to achieve and those that we wish to avoid ("consequentialism") rather than through some mindless repetition of an ideal that we believe to be sacred ("free-speech is so important"). Free-speech might be important but maintaining peace and achieving good outcomes for all involved should probably be held in higher regard [0] than some abstract principle, particularly when that abstract principle is a shibboleth of the people on one side of an issue but not the other.
[0] I'm talking about people that want to argue about racism and sexism while incidentally subjecting those they're arguing with to the most extreme and potentially toxic people in their group in the name of 'free speech'.