You've accidental confused achievement and ability.
A student who goes to a poor school who offers zero AP classes will always achieve lower weighted GPAs than students who go to wealthy schools that offer many AP classes. No amount of individual achievement will overcome that mathematical disadvatnage.
Racism comes into when, because of racism, it just so happens that schools who are well funded and can give that advantage are overwhelmingly white, and schools who are are poorly funded and cannot give that advantage are overwhelmingly non-white.
Suggesting that a student who is "placed above his or her station" into an elite program is setup for failure is a favorite of a few SCOTUS Justices, but in fact is not supported by evidence. Graduation and other metrics of success are not strongly correlated to past attainment, primarily because most programs have already had to deal with achievement inequities, and the most successful programs already have mechanisms to even out unequal prior achievement. Virtually all programs that have elite programs already have a substantial apparatus dedicated to filling achievement gaps between incoming students.
The larger question of "what is the purpose of college" and "is it to educate people" is bigger than this thread, but shouldn't be overlooked. For schools that are publicly funded in whole or part, there should be a larger mission than sending young adults through an educational meat grinder. The public mission of public universities should absolutely have a social justice component.
There is absolutely no doubt that we could design an education system which, at an early age, divides and tracks students towards a successful and high level of attainment. By ruthlessly focusing resources on those children with the best chances of success, and minimizing resources expended on those with lower changes of success, the system could produce many multiples of positive outcomes than we do now, but at the cost of many more left with almost no attainment. Our present distribution of resources, in the US, is haphazardly assembled and produces a balance of outcomes, but is by no means optimized for any particular set of outcomes.