I'm fighting against speech that dehumanizes Jews, LGBT, Blacks, and so on; and ensuring that people who engage in it face consequences for that speech. This gets back to Hannah Arendt's paradox of tolerance.
The conclusion his argument leads to is: In a tolerant society, attacking people because of their beliefs is not permitted (requisite intolerance).
You don't fight against speech, you speak against its ideas.