No, they fail at math because they have never learned how to use their brainpower effectively. The difference between geniuses and everyone else, from my experiences, are not any fundamental difference in their brain wiring, but that they learned how to better focus what brainpower they do have,
Its like the difference between a guy who gives 120% every day but his work, taken individually, is mediocre vs a guy who gives his 1% but his 1% is the Nobel prize winning work of his field... the guy who gives his 120% day in and day out is a smarter man.
Ironically, an IQ test will not say much about either of them. The 120% guy might score lower than the 1% man by 10 or 15 points, but the 120% guy, if he were, say, a researcher or inventor, would be spending day in and day out finding thousands of ways that don't work just so he can find one that does. The 120% man learned how to focus the skills he does have and learned how to apply them well; of the two, I'd hire him over the 1% man, Nobel or not.
The few super geniuses that history remembers (such as Tesla) had both: 120% work ethic combined with a brilliant mind. These people are rare, but every single person no matter their racial or religious background has the ability to become that kind of person to.... they just never learn how to.