Unfortunately, both yes and no. The word has become near profanity now and is often used as an insult rather than a statement.

The reaction of, "How is it racist?" can be taken as a defense even if it is an honest question.

Luckily, words are amazing and there are ways to diffuse this stuff. "I think some folks might find that insensitive, you know" might be better wording than "That was rather racist" - even though I might prefer the more direct version. Likewise, the response can be reworded for different effect as well.