In high school, I was one of those people who spent tons of time and energy on it, and failed miserably, barely graduating as a result. It wasn't until I was about 25 when I picked up a basic algebra and trigonometry book on a whim and started self-teaching, that it began making sense to me. I eventually went back to school for pure math.
In my case, I think what helped me as an adult was the freedom to learn on my own terms without the pressure of deadlines, and also a strong support network of friends encouraging me (compare with my teachers in high school who would actively embarrass me in front of the class or joke about their low expectations of me). And of course, some years of life experience in the real world doing things that were actually hard compared to (elementary) math as someone of limited means helped a bit too.
So, all of that is to say that people who struggle in math classes struggle for different reasons. I suspect that much of the time the dominating forces are psychosocial, cultural, or plain & simple lack of interest (which itself might be driven by traumatic experiences with shitty teachers or parents with weird expectations)