Well, the alternative is that merit is distributed in a racially uneven way. Is that your thesis?
That isn't the definition of "merit" that these colleges are looking for. Demonstrated academic performance is a part of it, to be sure, but it's not the ballgame.
e.g. https://college.harvard.edu/admissions/apply/what-we-look
Statistically, advantages along some dimensions are distributed unevenly across racial groups. That would predict differences in performance outcomes for members of racial groups within hierarchies where those advantages are selected for.
Personally, I don't think "ability to get into Harvard" is a particularly useful life skill, but I know opinions differ and I'm OK with that.