In fact, when it comes to government regulation of speech, the philosophical approach taken by the First Amendment, and the very thoughtful jurists who have interpreted it, is the opposite: that we must tolerate intolerance.
And as for the social sphere, where legal freedom of speech does not apply of course, the problem with too liberally being intolerant of "intolerance" is that the definition of "intolerance" is itself a political thing, inevitably expanded and wielded as a weapon against dissent. For example, opposition to more immigration becomes branded as inherently intolerant.