Tolerated is taking on a wide range is meanings here. On one the of the spectrum, absolutely no, part of a free society is not forcing people to tolerate speech they find intolerable. If I go to a TED talk about fixing poverty and the speaker talks about his plan for eugenics and a final solution, I will walk out. In the other end of the spectrum, the government may disappear people who's speech they or the society that votes them in intolerable.

You can be anywhere along that spectrum without it being some inevitable slide towards the government disappearing people. Freedom to associate is a form of expression.