This sounds a lot like the "Just World" fallacy under the cover of genetics. At one point 99% of the population was illiterate and living in thatch huts, and no doubt members of the elite at that time also considered this to be an expression of natural order as well.

But as it turns out, almost all of those peasants were capable of learning to read, doing algebra, engaging in abstract reasoning, labor coordination, and operating some fairly complex machinery. Many of them were even capable of becoming software engineers. We know this, because pretty much all of us came from them, and if we can do it, they could've done it, given the proper environment. Someone advancing your argument 1000, 100, or even 50 years ago would've wrong, what makes your argument different?

Of course we shouldn't expect equal outcomes from all people, but to suggest that we've now reached such a level of access and fairness in society that everyone still at the bottom can't possibly be elevated any further seems... unintelligent.