Is there any evidence that intelligence is genetic? I dont think your height was stopping you from getting to the NBA, nor is that even remotely obvious. You may find your height to be discouraging, but you could have very well possessed the potential to play professional basketball. You'd be 3 inches taller than Muggsy Bouges who played for 14 seasons, 1 inch taller than 13 year vet Earl Boykins, and the same height as slam dunk contest champion Spud Webb.
Colleges might be letting more people through, but I dont think people are any dumber. Some people just aren't concerned with understanding everything they come across. Some kids go to college for no other reason than that's what they're supposed to do. If they cared they would be able to learn just as much as you or anyone else.