I see "retarded", "Jew" and other terms (when used pejoratively) as in a separate usage class from "fuck", "shit" and their ilk. The former are invectives; that is, they refer to a specific
kind of person. Their usage can
only be interpreted as negative judgment of a human being, even if they're not specifically talking about someone. ("Don't let them Jew you" is still negatively judging Jews even if "them" is vague.)
On the other hand, "fuck", "shit" et al. exist in a bit of a vacuum. "What the fuck" is not offensive to fucks, because (hopefully) very few people identify as "fucks".
So, the vector of offense for "Jew" is different from the vector of offense for "fuck". I'm offended when people use racial slurs. But when someone simply swears, I don't see a reason to be offended. It may affect my opinion of them; for example, I am less likely to respect someone for being articulate if they can't express themselves without swearing every couple of words. But getting offended implies some degree of emotional investment in the words or concepts themselves, which I don't really see the basis for.
To put it another way: I feel like any reasonable, compassionate person can't help but be offended to some degree by slurs; and they should be, because slurs are probably not such a great thing. Conversely, I think that we are taught to be offended by swears, and it's within reach of the average person to disconnect their emotional response from them.