A much larger number of people attend college today than in the past. Since most of the smartest people were probably already going to college, increasing the size of the pool would almost certainly decrease the "average smartness" of college students.

Alternatively, you could just say that historically college students were disproportionately the "best students" during primary education, and increasing the size of the pool would result in more people who were not as good students attending college.