I'm white so I can only postulate from other experiences where I can actually relate.

I think the question of "black hat" may in fact be offensive is valid criticism of our culture using "black" to be on the opposite side of a spectrum compared to white; where humans think the white side is possibly noble while the black side deserves some form of contempt.

I think it's a red herring to think about other combinations of words having "black" before the following word and where no negative feeling exists from it. Example: black swan

Should people care? IDK