That's a very fair counterargument!
I think I do not see it as realistic that those things you mention are threatened by mass migration (whether of refugees or otherwise). As a concrete example, I live in Brooklyn, where there is a significant Jewish sub-community with their own cultural identity and in particular very conservative religious/political views. This community showed up in Brooklyn largely because they were part of the Jewish migration from eastern Europe in the early 20th century, escaping anti-Semitism there (so, this was not quite voluntary migration by people who chose America for American values). They're very successful in their own neighborhoods and absolutely maintain their own values, but don't seem to have had any effect on Brooklyn's liberal values or the culture at large; if anything we're one of the communities most stereotyped as liberal to a fault.
Usually when I see fears about cultural changes in European countries receiving migrants, that fear is expressed in racial or at most religious terms, and not in political terms. I see things like "your child will be going to school with Muslims," not "your child will be taught at school that the right to free speech doesn't cover blasphemy." Yes, you can argue in theory that the former could imply the latter. But that doesn't seem to be what's actually happening. (If it is, I'd appreciate being corrected!)
Also, at least as I see it, the lives that were lost fighting for legal and social equality of the sexes, worker's rights, etc. were almost universally not fights between European culture and non-European cultures. They were generally fights within a culture, and occasionally fights between different Western cultures (e.g., the US war for independence). I do not see people arguing against migration e.g. within the US or even between Western states, just from one particular racial/religious origin to any Western nation, whether or not that Western nation believes in legal equality of the sexes, socialized medicine, free speech, etc.