The problem is deciding who gets to define those concepts in edge cases. Otherwise motivated parties try to push everything they don't like into those categories. "Hate speech" is a common example.
Who gets to decide draw the line between the sincere desire for free speech, and extending the line to include disingenuous calls for "free speech" which are designed to shelter forms of toxic and manipulative behavior that people are just using as a tool to push their preferred ideology?