I think most of the difficulty is 1) determined by your expectation of it 2) initial foreignness and the invisibility of the gap between the concepts you need to understand something, and the ones you already have (i.e. when an outsider just looks at some mathematical statement they have no comprehension of, it's not clear that there are a few layers of concepts underlying it which could be smoothly traversed, as long as one puts the time into it and is given some direction.)
That assessment didn't come from nowhere.
Given that there seem to be aptitude differences for most physical things, I would find it hard to believe that there are no aptitude differences for most mental things, especially for something so unnatural.
Not to mention that all the things you speak of are very real difficulties to many people that they're not going to get any visible pathway through.
I'm not sure where you're getting the 'wide range of creatures, not humans' part, or even why that matters, but I agree, mathematics is a formalization/projection of our own thinking process, which pretty much every human has.
I suspect a more visual approach to be more useful than the classical textbook approach. It is true that a lot of textbooks do use pictures, but video's would help a lot, I think.
However, there are people who learn better algebraically than visually. So, combining different approaches would probably be optimal -- the problem currently is that often a very dry (and anti-historical) endless litany of definition, theorem, proof is taught.
I do agree that math can and should be widely appreciated.
You can pursue new mathematics without having a PhD. I think the most difficult/frustrating part about academic/institutional mathematics is the politics and bureaucratic bullshit.