It’s all appeal to authority or emotion, with some science going on, but mostly scientism.
edit: You can replace “intelligent folk” with “intelligentsia”, as to not imply any IQ difference. I’m contrasting “fringe communities” with the more mainstream “intelligentsia”. The difference at play is perceived intelligence, not actual.
Since I anticipate somebody appealing to the Flynn effect [0] or similar improvements in intelligence-testing scores across the recent centuries, I would head them off by mentioning Spearman's hypothesis [1]. The obvious reading of the hypothesis is horribly racist, but we can take its contrapositive: If there are largely no changes between humans in recent millennia, then the differences in testing scores are due to cultural biases in the construction of the test and the education of people. And it happens to be the case, due to pedigree collapse [2], that we can be mathematically confident in the homogeneity of humanity (to say nothing of confidence from molecular biology [3]).
[0] https://en.wikipedia.org/wiki/Flynn_effect
[1] https://en.wikipedia.org/wiki/Spearman%27s_hypothesis
Not being sarcastic, I suppose things you see in IQ tests like spacial manipulation are culturally biased in that certain cultures give kids shape-in-the-hole toys, teach them shapes, have geometry classes, etc. You could say improvements in intelligence scores are teaching to the test. You could also say we're standing on the shoulders of giants.