Your rationale is exactly what every dictator throughout history has used to protect their power. Every, single, one. You either have free speech and allow all to express their ideas unfettered, or you will end up in a dictatorship sooner or later, guaranteed. To be clear, i'm talking about expressing ideas like "men and women differ biologically" and not speaking about threats/fighting words. Once you start punishing people for their ideas you're on the road to dictatorship, at least if every dictatorship in history is any guide.