For those who aren't doing the math, there are now about 340M Americans, so 500K immigrants per year would be about 0.128% or roughly 1/780 of the population arriving as immigrants per year. If you figure the average immigrant stays for about 5 years (there's likely a long tail, where many people on temporary H1Bs go back to their home countries in 3-5 years while green card holders stay forever), you'd expect < 1% of the U.S. population to be immigrants under this system, and the average person would know maybe 1 or 2 immigrants.

FWIW, my community is 30% foreign-born and we don't really have any social cohesion problems. I think that length of time in the country actually matters more than the absolute number of immigrants; people who have been here since they were teens are very well acculturated. But if I were to put numbers on how many new people a group can absorb per year without radically changing its culture, I'd put it at about 10% (that's like having one new person on a 10 person team hired within the last year). If we extrapolated to the whole U.S. population and assumed they were all evenly distributed, this'd be 34M people/year.

Honestly, I think the bigger problem is that they're not evenly distributed. Some places in America never change, over generations, and remain very suspicious of outsiders. Other places become 80% immigrant over a span of 10-15 years, and then they never talk to anyone outside their ethnic background. In not very long, you end up with multiple different Americas that can't relate to each other. The parable of the polygons is very relevant here:

https://ncase.me/polygons/