According to J.S. Mill, free speech should only be constrained to the extent that it violates what he calls the harm principle:
"The only purpose for which power can be rightfully exercised over any member of a civilized community, against his will, is to prevent harm to others."
He makes a distinction between causing harm and causing offense. Revulsion, disgust, psychologically upsetting speech, discriminatory speech, insults and so forth fall under the latter category. Free speech which merely causes offense should not be prohibited according to Mill however controversial or contrary to social norms such speech is.
But how strong the definition of the "harm principle" should be is subject to debate.
Contrast for instance, screaming "Fire!!" in a theater - which might cause injury and death from a crowd stampede - with "All <insert_ethnic_group> are inherently inferior to <insert_other_ethnic_group>".
The difference between causing harm and offense is that harm is universally injurious, whereas as what causes offense is only experienced subjectively.
Although I find blatant hate speech detestable, there is a worrying trend where a majority of people consider hate speech to mean "anything that we disagree with". In our times, it is difficult to speak frankly or be a contrarian without being demonized. And this is precisely what J.S. Mill calls "tyranny of the majority" and consequentially what leads to loss of individuality in society.