I fundamentally disagree that you can't use a word in one context because of its use in another. If I call someone a bastard I do not mean that they were born out of wedlock the context and use of the word are entirely different.
That being said, given your example I'd ask you to consider this situation: If you call someone a bastard jokingly and they happened to be a child who was born out of wedlock and it was a sensitive subject for them, do you think it would potentially be hurtful? Is there another word you could choose to get the same point across that might have been less hurtful? If so, is there a significant downside to using that other word instead? If you found out that someone was literally a bastard and the word made them unhappy, would you continue to use it when talking with them?