In the former case, yes, natives are displaced and horrible things happen to them, but the key difference is that the colonizer really has to supply all the human labor. It means colonists of the colonizer's own culture come in and build a society similar to that of the colonizer. This probably also results in that kind of colony being treated better by the colonizer because they were "white" or "British". That's why basically only white countries are part of the British Commonwealth, whereas the nonwhite colonies have severed ties with the UK. Also, colonists in this scenario would largely be citizens of the colonizing country, and would demand commensurate rights and legal protections.
What seems to me a very strong pattern is that colonies in sparsely-populated regions turned out generally well and are first-world countries today, whereas countries that were heavily populated at the time of colonization/conquest are largely still basket cases.
I think the most likely explanation of this, besides cultural affinity with the colonizer, is that if you are colonizing a sparsely populated region, labor will be scarce. It won't be economical to have vast numbers of colonists extracting any but very high-value raw materials; or, more likely, processed raw materials and low-level industrial goods. So there would be much more pressure for such colonies to industrialize faster. And in turn, a colony with higher per-capita productivity will be able to develop a middle class and therefore a functioning domestic market much faster.