You can be opposed to whatever you like. Voicing your sentiment will, however, shine a light upon your understanding of the world. Maybe you could ask "Why do people think that I'm a bad person if I'm against mass immigration?" Assuming this comes closer to your request, I will try to answer that. But I will have to digress a bit, so please bear with me.
I have the feeling that how you see the world is very much characterised by whether you are (what I call) an inclusionist or an exclusionist.
Let me explain what I mean by that. Since we're on HN, I will use an analogy that might be up your alley. In CS, there is something called local search [1] for optimisation problems. The basic idea is: let's say, we have a number of states and we want to find the best one. And let's say, we have a method of determining how "good" a state is. A simple approach would be to start at some state, look around at the neighbour states and pick the better one. Then we move to that state and look around again. And so on, until we're at a state that is surrounded by only less or equally "good" states.
The problem with this method is that we might end up in a local maximum or on a plateau, but we're stuck there forever. To get past that, we can choose a random state now and then. This is what simulated annealing [2] does or the mutation in genetic algorithms [3]. Even though at times we might be distracted from the best state right in front of us by following serendipity, this strategy is overall superior to the myopic, greedy strategy.
This is an interesting insight: by including something into our consideration that is most likely worse than what we already have, we achieve an overall better strategy. This is not very intuitive. Let's keep that in mind.
This concept can be applied to (we're leaving the realm of science here, but being scientific wasn't the point in the first place) our world view in general. Let's call it the inclusionist view, while the opposite would be the exclusionist view.
The exclusionist would prefer everyone to stick to the generally accepted way of life, while the inclusionist sees the need to allow deviation from the norm.
Since the insight from above is not very intuitive, the inclusionist might think that whoever is for making an end to deviation and all for following the one true normative way must be a myopic exclusionist who is too single minded to grasp the bigger picture.
If you're in the inclusionist camp, you will have had a lot of frustrating conversations with exclusionists who will insist that going with the most likely less "good" option is stupid and you're a beady eyed, gullible philanthropist. While you might come to the conclusion that the exclusionist is a myopic, uneducated, neanderthal.
If you have witnessed many political discussions, a certain pattern emerges, and you get the impression that the racists, nazis, fascists, philistines are all the outcome of exclusionism.
Now, let's go back to the original topic. Immigration is one of those topics where the views of inclusionists and exclusionists play a role. Everything is kind of fine as long as the numbers are small and the places of origin for immigrants are not too outlandish. But when someone calls for caps or selective admittance or anything else to quickly deal with "mass immigration", the inclusionist smells an exclusionist. Someone who did not grasp, the inclusionist assumes, the greater ramifications of applying broad selections, etc. The inclusionist will also be wary of not allowing the exclusionist to make too much ground, because he suspects the same shortsightedness that goes with racism.
So, this is basically why you get a negative response if you use certain words or raise topics in a certain way. I have no idea, in which camp you are or if either camp actually exists, but maybe this might provide a glimpse into the slightly elitist mindset of oh so sophisticated people like me.
[1] https://en.wikipedia.org/wiki/Local_search_(optimization)
[2] https://en.wikipedia.org/wiki/Simulated_annealing
[3] https://en.wikipedia.org/wiki/Genetic_algorithm