Humans are all unique. At the same time humans don't vary that much. Compare with some other species (dogs are often used) where there is a wide range of size, intelligence, and any kind of thing humans might have bred dogs to do.
We are a social species. We pay lots and lots of attention to one another. We constantly compare people. That kind of scrutiny magnifies differences.
Add to this that we're pattern-manufacturing machines. If we stare at things that are ostensibly the same for long enough, we'll find some way to classify and order them.
> Would it be a good idea to make society more inclusive for the less fortunate, intelligence wise?
Why not take the opposite approach and teach that like any other human differences, they deserve to be celebrated, but like any other human differences, they're not really so different?