In Turkish there is a very fitting proverb which loosely translates to "Whoever has a scar will take offense" meaning if you say something, and someone overreacts to it, it's not your fault and probably just because they have some hidden problem or feeling of inadequacy (the "scar").
If everyone can take themselves a little less seriously, we'd all be a little less pissed. Thank you for listening. </rant>
Edit: Okay, here comes Pandora's Box again. The way I see it, what is so offensive about racist speech (or any generalization for that matter) is the presumption that all members of that group are equally <insert derogatory adjective here>. Once you realize how absurd, dumb and wrong that is, I don't see how anyone could possibly take racism seriously and be offended by it. At most, you should conclude that the person who is making the remarks is probably not intelligent, and not worth your time.
Heck, perhaps I have some sort of emotional deficiency or something, but I don't even see how people can get so angry about words. For example, I have seen people taunting each other with stuff like "your mother was amazing in bed last night", and the other person getting gradually angrier up to the point that they start physically fighting. I think it's a very curious thing about the human psyche that I can make up something absurd, blabber about it, and get another human to the point of total lack of self control. I think it's all just a dumb made up social convention that we're better off without.