Sure, I agree everything has nuance and context to it. I was really just stating the rationale I've seen given for avoiding those words rather than offering my opinion.
That being said, given your example I'd ask you to consider this situation: If you call someone a bastard jokingly and they happened to be a child who was born out of wedlock and it was a sensitive subject for them, do you think it would potentially be hurtful? Is there another word you could choose to get the same point across that might have been less hurtful? If so, is there a significant downside to using that other word instead? If you found out that someone was literally a bastard and the word made them unhappy, would you continue to use it when talking with them?