I distinctly remember a professor in college saying that he felt like the caliber of students enrolling every year declines. I was kind of offended, but hey maybe we was right.
Alternatively, you could just say that historically college students were disproportionately the "best students" during primary education, and increasing the size of the pool would result in more people who were not as good students attending college.