There's no such thing as "jobs natives don't want", only "wages natives don't want".
> Maybe, or maybe they just employ less people and less things get built.
The foreign born population residing in the US fell from ~15% to ~5% from 1920-1970. It was one of the most prosperous and productive periods in human history.