I'm not sure what the correlation is between hard work and success, but it's much weaker than at least ten other factors.
In much the same way, you can bust your ass working as a cart fetcher at a grocery store, but unless you are intentional about how you are going to use that hard work to leverage yourself into a better position, you are just going to maintain.
Hard work should always be balanced by some amount of self education and intention, or else you run the risk of treading water and getting nowhere.
The reality is that learning is an uncomfortable process. A lot of people find a comfortable rut and they seem to prefer just staying in it. This is why public education is so important. It forces people to give themselves at least a bare minimum amount of education. Just imagine how dumb and poor everyone would end up if parents weren't legally required to educate their kids.
The fact that you think people who failed must be stupid or slow learners says a lot about your worldview.
I mean... it's not rocket science.
It's a complete cop-out to say that only young men with no obligations can do such a thing. My parents had tons of obligations and responsibilities, and they found a way to move when necessary.
People get way too attached to the idea of staying in one place their whole lives. The world is a big place.