I think there's a difference between the sort of high-skill and/or wealthy immigrants that create billion dollar tech companies and the low-skill immigrants that americans are mostly mad about.
This nuance is important.
Perhaps for you the distinction is important, but in terms of policy, there is no difference at all.
If you start kicking out all the people who are brown or muslim or otherwise different, you hurt the society and make it harder for the legal high-skilled immigrants to live here.
(Not to mention that for three decades American politicians and corporations encouraged immigrants to come here and work without status. And that Irish and Italian immigrants came here 100 years ago before there was any immigration law; if today's republicans existed in 1900-1930, they would have demonized Irish and Italians for votes too.)
As a brown person it's become abundantly clear that the Stephen Miller, Steve Bannon view on immigration is certainly mad about something - and it's not skill.