You have it backwards, the "no barriers" mainly came from being wealthy. I'll give a small anecdote: One of my friends is from a very wealthy family. He is a person of color. He was still able to gain admission to an elite boarding school, now works for a prestigious company, is respected by his peers, etc.

Privilege mainly comes from wealth. Of course it is a serious problem in society when certain groups have disproportionate poverty or less wealth, because that indicates some structural cause.

However in the modern day that cause is simply the fact that most people, regardless of skin color, do not move out of their parents' income decile. And when one group is already more economically disenfranchised than other groups, then that economic disenfranchisement will perpetuate itself.