Not only that, the opposite doesn't. There are fewer black kids than white kids who grew up in affluent / middle class families, but not so few that you couldn't fill every diversity slot at an elite university with them.
Which defeats the proper purpose of the program. You're supposed to be giving opportunities to people who didn't have them and bringing different perspectives into the student body, not further advantaging the most privileged subset of the people who can check the important box on the form who grew up in the same neighborhood as the other elite students.