As possibly a tradeoff for this intense visual imagination, I have dyscalculia: I am unable to do more than simple calculations in my mind, and the more math I do the more my mind "closes up" and I feel a kind of pain and visceral frustration. There was nothing more frustrating growing up than people telling me I just needed to be patient or try harder, when nothing I did had any effect on my math ability.
As an adult I have worked around that disability by learning the fields of math that don't involve lots of numbers (discrete math is fine, algorithms are fine), and by learning non-numerical strategies for breaking down problems into smaller, more manageable calculations.
People are different, we have different cognitive abilities. Just like some people are blind or deaf, some people lack a visual imagination, an internal narrative, or the ability to manipulate numbers. I think this cognitive diversity is really quite fascinating.