My impression is that IQ seems to be a measurement of a person's capacity to think abstractly, which is something you learn through the world around you. Perhaps those who fail at math have simply never experienced anything that requires abstract thought. Therefore, it seems like there is a skill that even basic arithmetic is dependent upon, basic abstractions. I can see how an otherwise able person would be hopeless at math if they never learned abstract thought.