The part where I disagree with you is the notion that there's something xenophopic about feeling that current levels (depending on where you get your estimates, this ranges from about 1 to 1.8 million per year) are a reasonable limit. I do find it surprising that a nation that admits well over a million immigrants a year is so frequently accused of "anti-immigrant" policies because the annual cap isn't even higher.
You say you'd increase the quota, but you didn't mention if you'd eliminate it altogether. Suppose you felt that immigration levels should be double what they are now, say about 2.5 million a year. Does that mean someone who would make it 5 million a year can call you anti-immigrant?