Yale, Columbia, Community College and a State school here. I've also taught college courses at an extension school of a public university.

The population differences are real. They're like normal distributions where the mean of one is perhaps a whole SD higher than the mean of another. That said, if you pick the top 5 students at the State School and the top 5 at the Ivy, the students themselves are roughly equivalent, but the ones from the Ivy had a lot more access to higher quality facilities, funding, opportunity, faculty, and peers.