I have a Ph.D. in physics, and I might consider myself below average among physics Ph.D.s coming out of Harvard or Berkeley, but am I "below average" in physics without further qualifications? It would be silly to say that, wouldn't it?
Like the mathematical genius saying, "I am not that talented, I can't even cook, my spouse has to tell me five times a day where I left the keys (haha), I simply loved math more than others, but anyone, with the right tenacity and guidance, could accomplish what I was fortunate enough to achieve". Come on, those are not serious people.
It is likely that my expertise gives me a more nuanced understanding of the complexities of the field, of how much there is to learn, than the understanding that a talented high school student should have of it -- I agree with you on that -- but a sober assessment of one's abilities and position within a population has never been a mortal sin, quite the contrary. It is the stuff that serious people are made of.