I have a brother. Due to circumstances, I can call him a "stupid idiot" and he will know that I'm not serious and I respect him. However if a stranger calls him the same thing, he will take offense. Is that an unfair double standard?
Similarly, the n-word is a derogatory racial slur. Black people are quite justified in taking offense when being called that by an non-black person. However, if called that by a black person, the listener can be reasonably sure that the speaker is not being racist, and also that no 3rd party listener would mistake it for that.
It's just an obvious fact of language that context matters, and the speaker matters.