I think this whole debate is centered on the shallow idea that "going to an Ivy League" in and of itself is what ordains "elite-ness". To unravel this, you first must consider the actual value proposition of Ivy Leagues relative to other schools.

Firstly, undergrad in America, especially at private schools like the Ivy Leagues, is more a daycare for rich kids, than it is a place of scholarship and learning. The education is certainly good, but there are plenty of other schools with rigorous academics. At these schools the children of the rich fraternize, drink and come of age with other children of the rich. The schools themselves are well connected as well but it's in large part through these bonds with their rich and powerful peers that the graduates of these schools go on to become "the elites". If you reduce the number of kids from rich and powerful families going to these places, you are going to drastically reduce the networking potential amongst classmates and start to erode school level affiliations. In short, you can't just limit admissions to solely "academic merit" without destroying a big part of what makes prestigious schools valuable. Ultimately, if the children of the rich and powerful are excluded, they will congregate elsewhere and all the prestige and kingmakers will follow.