In the real world, there are winners and losers. People show up in a new place and challenge the current inhabitant's claim to possession. It's always been that way. There's no point fretting over a war that was lost 8-10 generations back. Today, the descendants of the European settlers have as much if not more rightful claim to the land as the descendants of the Aboriginal Americans.
All power eventually reduces down to force. If you have the ability to enforce your possession, congratulations, it's your land. This may not comport with the fairy tales that even grown-ups cling to today, but it's how the real world of scarcity, competition, and meritocracy works.
Immigration is good, but if there wasn't an interest in ensuring that immigration policy benefited the host nation, immigration policy wouldn't be a thing. No foreign national is owed access to any country.
Some immigrants are productive and beneficial to the host nation, and some aren't. Immigration policy attempts to filter the wheat from the chaff and to ensure that in aggregate, the citizens of the host nation are benefited, not negatively impacted, by the incoming foreign nationals. That's not a controversial statement -- it's the entire theory of immigration policy in the first place.