It's not just working hard enough, but also doing the right kind of work. Many people make the mistake of trying to memorize things without understanding. Which may be easy at the beginning when you memorize a fact or two, but it gradually accumulates, especially in math when the old topics never go away as the new ones are introduced. And then the memorizers are actually working much harder, and even that is not enough, so they fail.

So why the aversion to understanding? I suspect part of that is generational; if your parents sucked at math because they relied on memorization, they probably won't introduce you to math as an something worth understanding. It will either be "give up", or "work harder" but in the sense of memorizing harder. Not just your parents, but the entire culture around you will be like that. Another part is that most math teachers at elementary schools actually suck at math; because teachers are many, but people good at math are few and they have many better careers available. But another problem is the insistence of school system on everyone going forward at a predetermined speed -- sometimes understanding takes time, and when you don't have the time, you are forced to memorize; but once you start memorizing, you usually need to keep memorizing, because understanding can only be built on understanding the prerequisites.

Properly taught elementary-school math should be fun, like this: https://www.matika.in/en/ Fun makes people think.