As you look to populations who didn't go through the same kind of education, social circles, family environments ("privilege" as they say nowadays), you find people who:
-- Don't hold conversations with the same kind of productive interaction
-- Don't have the same goals of success or motivations (individually or team)
-- Haven't yet been exposed to professionally challenging situations and know how to handle them healthily
-- Have a higher quotient of "fuck it" if they encounter difficulty
-- Display gaps in judgement on how to behave / what to do or not do at work
-- Just have more hard-to-manage stresses outside of work on their life
-- many more personality / work-related traits
Of course I'm not saying that everyone from a less privileged background is like this. But undeniably as you "turn the knob down", you encounter more and more of these issues. If it were not true, everyone pretty much would want to hire from every stratum of society.
It's a sad truth that the indicators of a privileged background aren't just superficial meaningless markers. They actually do pretty well indicate how likely someone is to succeed. They mean that someone has gone through education / upbringing that give you a pretty good sense of how that person will behave, and perform (in general). Hence the vicious circle.
This is why efforts like at Accenture (?) to strip resumes of university background or other "biased" indicators will be unsuccessful. The noise-to-signal that results will be overwhelming.