At any of the most elite universities there are many more qualified applicants than there are spots. For example, Harvard could fill its undergrad ranks with valedictorians alone, twice over. Thus, the fair thing to do would be to either (a) drastically increase the size of the incoming classes, or (b) set a minimum level of objective criteria (e.g. test scores, class standing, etc.) and then just run a lottery from those qualified students.
But instead, the admissions offices at schools run a system much worse than random. It's a system that's highly subject to pay-to-play access (and usually not the outright fraud kind, but more the endow-a-chair kind) while at the same time allowing the universities to keep up the charade that it is all merit based. Even if there weren't undo influence, the current efforts to maintain "diversity" (and not just the racial kind, but the kind where we need 5 cello players and 7 chess club champions, etc) means the admissions criteria is already completely arbitrary if you meet the basic (albeit still tough) academic requirements.
The reason this won't change, though, is the primary purpose of today's elite universities is to maintain social stratification. If universities did the fair thing and had a lottery for all qualified applicants, it would mean they would have to admit that they are not the objective arbiters of "potential" they purport to be.