For free speech to be a reasonable thing to priorise over other things you have to have some more fundamental rules of society in place. E.g. fighting for free speech during the rise of the nazi party in 1920s Germany might sound like a noble thing, but once the Nazis whos speech you defended are in power they will shoot you at your doorstep for defending that of their enemies.

That doesn't mean you have forbid Nazis from ever speaking (as if this would work), it means you have to make sure your democracy and your institutions are stable enough to survive their first period in government and manage to get a peaceful and fair transfer of power after that (instead of Nazis setting fire to the Reichstag, blaming it on communists, declaring an emergency and never have elections again).

I'd rather loose the right to utter certain inflamatory phrases than loose the right and the possibility to vote the next government out of office in a fair and representative election.