Good point. If you let the government silence extremists then they can also do it to the moderates next. Those are subjective definitions after all.

Paradox of of tolerance makes it clear that what can't be tolerated is anyone promoting law that restricts or hinders freedoms of religion, speech, press, assembly, and petition. Lest we be left with no reasonable tolerance for any moderate speech at all, and we're left with intolerance toward all speech. Fortunately the first amendment exists and we don't have to worry too much.

Thanks for pointing that out though, it's a great talking point for free speech absolutists.