From my personal experience: If one is able to dig deeply enough, for what I guess to be something like 90% of the population, there is a level where people just declare "I feel that way" and there is no logic, nor even the barest respect for rationality. What's more, these feelings are very often used to justify the arbitrary bad treatment of others who are deemed to be "wrong" or "inferior."

All of these feelings have some "rationalization" as window dressing, but ultimately, it comes down to irrational feelings of tribal affiliation. This invariably comes out in politics and professional life.

We're goddamned grunting, screaming apes who have the ability to occasionally surpass our primitive nature and are well suited to being organized into groups. The vast majority of us do not have the intellectual integrity to hold consistently to a set of rational principles to the nth degree. In the end, our feelings hold sway, and our empathy often fails.

Prejudice is a part of our very nature, and it comes out all the time. There is no need for it to be attached to race or gender, and our society doesn't equip us with the intellectual tools to recognize it at the substrate level. Rather, our present societies are organized around it.