"idiot" is not a synonym for the word in question here. This is similar to saying that "t
*ny" and "transsexual" are synonyms. They're not.
Let me give you an example of how referring to you in an insulting way can change your attitudes and discourage conversation.
1. Words have a lesser impact on you because you've [a lucky person] who doesn't live a life were they have been used to deny you opportunity and basic humanity constantly.
2. Words have a lesser impact on you because you're [a white male] who doesn't live a life where they have been used to deny you opportunity and basic humanity constantly.
Looking around the internet, #2 certainly seems more contentious. But let's be honest here, the overlap of case 1 and case 2 is much more likely in most of the western world than other ethnicity and gender and sexual orientation we could put in there.