I just found the question interesting. And it's a common phenomenon. Musicians, for example, even not particularly gifted ones, get this kind of question pretty frequently--"can adults really learn to be good musicians?", "it just comes naturally to you, right?", etc. Humans have funny logic sometimes, I guess, including the people on the receiving end of the questions.
That being said, mathematicians definitely have (as a class) some good mental attributes - they usually think very consistently and can see very subtle flaws in arguments (especially common ones (it gets frustrating)). They also solve really tough problems which means that their ideas about problem solving (probably pretty central to the question of "what is intelligence?") are probably good, assuming they're self-reflective.