Naturally this empathy only self-reinforces itself, if it selectively ignores those who agitate against it. Eg if someone advocates against free-speech, maybe protecting them won't preserve this basic empathy.
And it seems this instability is real, as we have a hard time defining the minimums of free speech.