That's an interesting point, however I disagree it's tolerance of one group that makes someone intolerant of another group. In other words, I don't think being tolerant of black people means I must be intolerant of white people.
Karl Popper wrote about this idea called the paradox of tolerance [0] asking if you should be tolerant of intolerance.