Got an related idea from the IQ thread: is there a correlation between aphantasism and IQ? Most IQ tests require you to visually imagine the sequences that lead to the correct answer.
I only found out about aphantasia early 2018. I had always suspected there was something different/difficent and went to a university study, and just reaffirmed my IQ and discovered that I reallly get slow on some tasks - I find the answer fast enough but I have a lifetime of distrust on my processes, so just out of habit do double or triple checks (which dont really make better answers, just verify what I am writing is what was meant, I guess). Of course, no one knew about aphantasia at this point.
I failed geometry in jr.high hard. It was almost impossible. But making and passing calculus was relatively easy (long story short: parents moved, lied about passing geometry).
I have mild dyslexia, so I think that mixed with the other just makes it reallly hard.
I think you assume it is required to do it visually only because you do it that way. Personally I've had no trouble with that kind of IQ test without using visualization.