I'm basing it on both my time as a math tutor for Pima Community College and on people I have known.
People like my mother who, if she can quantify the data she can effectively do algebra, but as soon as X and Y appear she shuts down. She cannot make the jump to the abstract thinking of math, and it curtailed her ability to go back to college. It didn't help that her math teacher in 9th grade told her that she would never be good at math, much like many women from poor backgrounds. (I have no study, only the experience in tutoring on how many women said that a teacher told them not to bother, and could never make it over that hump.) Now you say that it's merely psychological for them and that they could. I'm telling you that for as much work as some of these people I tutored put in, they had a mental block that they simply could not overcome, fighting their way to a C in college algebra so they could get to where they're going.
I can give as an example my boyfriend, who can speak and teach two languages better than most people here can in their native language, and is a promising Ph.D candidate in his program. He worked for a month straight (with my help) to raise his GRE math to a minimum score. His limit may be calculus, but certainly not higher math, and not upper level sciences. Yet I have seen this man wake up, read, write, sleep, and repeat for weeks straight.
He's a gifted writer and academic, but even if he could muddle through (say...) managed information systems, he'd never be more than mediocre because his brain simply does not work that way. (He still asks for help with his Mac.) Square peg, round hole.
It doesn't benefit those who have a passion for the sciences to put him and dozens of others in the same class, wearing down the professor because they struggle to grasp concepts that future scientists understood in fifth grade.