When you have well off parents, you can literally be sent to prep schools which drill for these tests.
Pretty much all French physics Nobel Prize and Field Medal laureate when to the same top school.
https://en.wikipedia.org/wiki/List_of_École_normale_supérieu...
This is the point of training: the more training you have access to, the better you do. If it was not the case, then the notion of school itself as a way of training people to be able to think by themselves will not have any sense.
And that is just training. Even with the same amount of class hour, kids who don't have to worry about take care of their siblings, of the house chores, or of even having access to decent relaxing conditions will get higher score even if they are in fact less smart.
If you want to build an elite sport team, I don't think you want to artificially put less athletic kids for the reason they had to work harder.
I think the question is why do we need elite higher education at all. Maybe we don't. In my view, we want to funnel the brightest people there and make sure they get access to the best resources.
You are saying that you don't want less athletic kids being accepted artificially. That's exactly the point: the score does not correspond to the talent, you have to correct for it: to compare 2 persons on their merit if they have had different training, you need to calibrate to get a variable that correspond to the merit.