I attended undergrad at a big state school. My classmates were largely representative of the top 30% of my high school - people who are either gifted with raw intelligence, or who do well by working hard. My school had good sports teams, and the student body spent a lot of time and effort cheering for the teams, partying, and generally having a good time. I certainly enjoyed myself, though the academic standards were so lax that I was able to pass an entire semester where I attended no classes other than the exams. There are a lot of people who just aren't interested in the academic side of school.
Some years later, I went to a much more prestigious school for my MBA. This school didn't really have an athletic department (as a Division III school they were not allowed to issue athletic scholarships) and the student body as a whole was both smart and hard-working. I mean, sure, they drank their share too -- but learning was considered cool and group projects (while drinking, of course!) were a perfectly acceptable way to spend a Friday or Saturday night.
That attitude is infectious and it sticks with you. Once there is positive peer pressure to learn, suddenly everyone becomes invested in everyone else's success. You go from "I heard that class was an easy A so I'm gonna take it" to "I think that class looks really interesting. Want to take it with me?" Extrapolate that from school to life and you start to see why graduates of these elite schools on the whole tend to be more successful.