http://blog.case.edu/think/2012/10/30/empathy_represses_anal...
There's a long pop-science suspicion of this, dating back to Jungian psychology (where Thinking vs. Feeling was considered a primitive dichotomy). It's enshrined in many of our stereotypes about socially awkward nerds, and also in the structure of the tech industry (which has separate departments for sales vs. engineering, and common wisdom that you need a hustler & a hacker for startups).
This seems much more likely, as an explanation, than ideas that poor math performance may be transmitted socially, although of course it'd need to be rigorously tested in an experiment to be proven.