1) "We" have built something great and unique, and we want others to be able to partake of and contribute to it, but the greatness and uniqueness of it is special and we want to preserve that. This is the more conservative view.
2) "We" have an environment where great things can happen, and so we want to bring people into it and add their greatness to it to make it even greater. There is nothing special about what we have, nothing that warrants preserving, and in fact change - constant, disruptive change - is naturally good. This is the more liberal view.
It bothers me that (1) is often conflated with racism, admittedly (the bothering part) because it's something I happen to agree with despite being mostly liberal on other issues. I DO think we here in America have a culture worth preserving, and while I want to open it up as much as possible to the world, I won't support doing so to the detriment and/or dissolution of what we have. I don't think constant disruption is naturally good; I didn't when I was younger and I don't think it is now.
But anything even approaching a whiff of something like "preservation" is immediately labeled racist dog-whistling (which seems insane to me). Any kind of "hey, maybe non-Americans are trying to exploit this and that's not good" is racism. Saying that perhaps as much diversity as possible even if just for diversity's sake in all things is maybe not the best thing - again, racism.
We need to recognize that there are real, non-racist and completely rational and informed reasons why people are uncomfortable with things like H1-B abuse, and that we don't all agree on what immigration should be or accomplish.