I think this is misunderstanding of cause and effect, and correlation does not imply causation. High math skill does correlate with nerdiness, and ceteris paribus, nerdiness does degrade your life chances or overall happiness. It does not mean high math skill alone, if you are not nerdy, will degrade your life chances, but well, they do not have high math skill to understand that. :)