I think it has something to do with the cognitive load of assimilating language. A lot of what makes math hard is the language of math. Those that understand math find the language to be the easiest part and by far the most mechanical and simple to understand. Yet not everyone can pick up new languages. Most people as I understand it only have room in their heads for one or two. I would wager those that learn math or programming might have room for a little more. To learn a language is rote excercise, but not everyone can retain it for some reason afaik.
I think most people can learn what a "variable" is and might understand assigning a number or a "string" of letters. Yet even understanding arrays, hashes, and such low-level primitives is difficult. Going beyond that into functions, operators, generators, classes, objects, closures, data structures, memory allocation, pointers, references, etc... forget it. It takes a certain kind of person to be able to assimilate all of that and use it effectively.
Why else do we continue to invent languages "for people who cannot program?"
It's hard for a master to think like a student.