Freedom of speech wasn't granted to us to talk about weather while drinking coffee. Freedom of speech was granted to us to talk about difficult topics quite often when we don't like discussing it in the first place.

This paraphrase of a quote is often attributed to both Churchill and Jefferson.

The whole idea is that people who you are uncomfortable listening to have the right to speech right in your face even when it makes you extremely uncomfortable. The whole current political system is morally bankrupt when some of us are granted this freedom (radical immams) while others aren't (i.e. racists). Because then regular people on the street stop believing and voting for liberal democrats. Liberal democrats criticizing freedom of speech (not granted to racists would seem to make you happy) and democratic process (i.e. electoral college, brexit) -- they start looking like they don't care about their own values (like freedom of speech and democracy). You can't call yourself liberal and then be anti-free speech to this or another group or perceived group. This makes you look as if you don't understand the words you use to describe ideas you believe in it. You are not liberal if you ban freedom of speech to racist. Period, and of story. Look up liberal in dictionary.

This is one of the reasons why people vote Trump or AfD or Social Democrats in Sweden. They recognize mainstream political parties as anti-democratic (ignoring EU referendums) and anti-liberal (opposition to freedom of speech). So if all the political spectrum in anti-liberal and anti-democratic why not to vote for a party that at least isn't hypocritical about it?