Unlimited tolerance must not be extended to the aggressively intolerant, because this will destroy the unlimited tolerance. Central philosophical principle of free society. Karl Popper.
That is, per Popper, tolerating people who will physically harm you as part of the discussion means a discussion cannot be meaningfully held at all.
But people really like to stretch this to whatever edge-case meaning of "intolerant" would be convenient to them at the time...