You don't need to be a mathematician though. Many Haskellers I've worked with either didn't go to university or went for something else. I only ever learned C before Haskell, for instance.
Maybe this makes sense once you are very familiar with a library (like for example if you are a library dev), but if you are trying to learn a new library it makes for pretty much unreadable code.
For example
div <$> (Just 5) <star> (Just 7)
Would be pronounced "map div over Just 5 and Just 7".
I'd say learning Haskell caused me to read code without using words.
If you've written shell scripts before, you understand how Haskell functions calls are written.
If you've used `forEach()` in imperative languages then you understand functors.
These two things alone are half of the groundwork you need to read basic Haskell code! Of course, the fundamental underpinnings of the language will not appear just from a surface reading of the code, but if you're a person who learns by example then the above two are more than good enough.
If you're more of a linear learner then Learn You A Haskell [1] is an excellent, if earnest, introduction to the language. Depending on your confidence level you may be able to skip a decent way in.
You understand a single (or a class of) functor but I don't think it means you understand functors in general.
You also definitely don't understand how functors fit into Haskell's type hierarchy — its relation to monoids, monads, etc.
I'm only saying in relation to the idea of "reading Haskell code", it is possible to understand how they work (at the shallowest level) by analogy to `forEach` and to shell syntax.
Shell syntax also doesn't explain important constructs like currying, partial application, etc. But it's good enough that someone can read "boring" haskell code without being intimidated and turned off.
I write Haskell every day, professionally.
I'd recommend reading to someone who has a deep understanding of programming like John Carmack.