I feel like the left has historically been simply pro immigration, and then somehow race was brought into the conversation. I won't name the party I think responsible.
Simply false. Historically, US labor unions were anti-immigration because of competition for their own members already in the country. Go look up Cesar Chavez's view on allowing more Mexicans (especially illegal aliens) in to compete against his United Farm Workers. Unions being for unlimited, uncontrolled entry from outside the US is a very recent phenomenon.