In the long run they're still way ahead because they exploited those "low-quality" people for centuries, extracting all their labor and material resources for virtually nothing.
It also wasn't "centuries". The British Crown ruled India (which at the time contained Pakistan and Bangladesh) for less than a century, for instance.