I'm going to leave aside the issue of high-skilled immigration for moment and talk about Sam's addendum of "immigration more generally."
Low-skilled immigrants from Africa and Latin America have not yet shown evidence of assimilating into the American mainstream. Income and educational attainment is still much lower, the communities are still segregated, etc.
So there are two possibilities:
Possibility 1) It will be just a matter of time before they assimilate, just as it took time for the Irish or Italians to assimilate.
Possibility 2) For a variety of reasons, assimilation across racial lines is much more difficult. It is even more so when there is a fluid border with Mexico, and the U.S. no longer does the same sort of cultural imperialism it did in the early 1900's in order to assimilate immigrants. Thus long term, assimilation will not occur. Long term, the U.S. will end up looking a lot like Brazil, with a white overclass, a brown helot class, and a black underclass. Technology growth will slow as it becomes low status for the educated white overclass to do real work. Political corruption will grow as voters will vote as tribal blocks, and votes are determined by buying the votes, rather than making good policies for the nation.
I don't want to get into an argument about whether 1) or 2) is more likely. But I will argue that there is at least a 5% chance that 2) is true. If 2) is true, it would fundamentally change the character of the U.S. in a very negative way.
If we slow down immigration, and 1) turns out to be right, then we can always let more immigrants in later. If we accelerate immigration, and 2) turns out to be true, it cannot be undone. Not ever. Thus even if there is a 5% chance that 2) is right, it makes sense to slow down third-world immigration, until we can prove that assimilation will occur.