So it's not that sports fans are stupid. It's just that non-stupid people aren't so interested in sports. Got it.
There has been and still is something of a shunning of sports among the "intelligentsia". There have been studies about it, and how people who wish to be perceived as smart tend to avoid the things viewed as common. That still continues but I think it's reducing in frequency at least with respect to sports.
I work with a lot of really smart people. A big chunk of them are invested in sports and discuss games at work, wear jerseys on Friday, etc.
Also, if the average person is really interested in sports, then generally rising IQ wouldn't change the number of people interested in sports. The number of average people is still the same when everyone's IQ rises.