It is certainly strange and frustrating.
Being academically talented is generally socially frowned upon. On the contrary, being talented in other dimensions such as sports or music is generally celebrated.
In the general sense, there seems to be a confluence of factors at hand - I see it as academically talented people having not having great soft skills, academic achievement being of valued low by society, and a base aversion toward people of academic ability.
If you accept PG's explanation of academically intelligent people being deficient in social skills[0], then it follows that it is possible many academically skilled people have also not learned how to gracefully acknowledge their abilities (e.g. they're not modest).
From anecdotal personal experience it seems to me this is driven by two interrelated sub factors - the first is a lack of popularity / social value causing overcompensation. School years can be brutal for certain archetypes and this produces damaged egos which can result in certain unpopular behaviour in later life such as immodesty. The other is more tenuous but I have also found that progressing deeper along the thought process for STEM fields is at ends with social interaction - one requires absolute logical and rational thought, the other involves ambiguous emotional and often contradictory thought processes - e.g. in programming you develop a thicker skin toward being objectively wrong but pointing this out to people from other backgrounds can seem obtuse at the best of times, smart/show-offy at the worst of times. On the social sciences side, things are more relatable but only certain types of people enjoy engaging in "intellectual" conversation. This feeds into the last point.
Socially it is more glamorous to be a sporting champion than a mathematician. This is potentially due to relatability - it is easier to appreciate a good footballer or tennis player on a primal level than a brilliant mathematician or physicist: the sportsman's abilities are more tangible than the academic's. Its hard to sell academic work, it is by its nature difficult and the people involved do not spend enough time explaining this to people in the field to laypeople.
Finally, while we're not really all forced to do sports, we are all forced to do school and study maths, science etc. So for people who are weak in these subjects, it is natural to project the resentment held against the subjects to people associated with them. It probably doesn't helpe that success at school is a decent predictor for later life success - no one likes being reminded of their own inadequacy. It also doesn't help that things which are encouraged by academic activities like correctness and depth can be at ends with social interraction. Use of overly correct punctuation and grammar can seem formal and stuffy to normal people. Appreciation of arts and classical music can seem pretentious or posh. These things can lead to further alienation when passion is expressed in the wrong context.
It's worth noting that in East Asian countries where academic ability is more socially recognised, there seems to also be less of a stigma being brilliant or behaviour of people parading their inability at maths. Singapore is probably the closest place you'll get to having a nation that celebrates its great minds but its still a far cry from reaching the heights of popularity celebrities achieve...
[0]http://www.paulgraham.com/nerds.html