The unfortunate thing may be that we've conflated college admissions with this notion of "merit", rather than what you can get out of college itself.
Of course, I'm sure the athletics department does apply other forms of pressure to various parts of Harvard, making it easier for athletes to get admitted and pass their classes etc. But any scholarships they award are (corruption aside) at least theoretically based around financial need.
(Since it's hard to tell in text: no I'm not being sarcastic.)
I agree with most of your comment, but I think you're using "harder" and "rarer" interchangeably here.
One would guess that receiving a scholarship for elite athleticism or superior intellect are both quite difficult, and I have zero basis of comparison for which one is harder. I would not assume that athletic scholarships being rarer reflects on their difficulty. Rather, it's most likely a reflection on what the main function of a university is, as a place of learning. They admit more students to study than to play sports.