Maybe all non-European immigrants were only brought in as workers (and also slaves), so your argument may be right.
Let me rephrase then: The US has always been an immigration destination, so it's interesting that some immigrants are seen as different enough to be singled out. Spanish-speaking immigrants and Chinese arrived in US america much earlier than the famous NY around the turn of 20th century ships from Europe with Italians, Germans, Irish, Russians did.
I don't have numbers to cite, so I'll stop arguing more :).