It's true people don't have to "do" anything, but they are still affected by QOL decline: increase in traffic, not enough parking, schools get crowded, everything gets crowded, etc. So it's not a no-op.
If these are REALLY the objections, we could find ways to mitigate these issues.
A reasonable discussion could be had about how fast neighborhoods should change, or what reasonable expectations buyers should have about the neighborhood changing over what period of time around them.
However, if more people move in, their single family home in the period of 10-20 years will probably double in value and they'd probably be able to sell it and move to the outskirts of the now-expanded city with a similar character to what they had before, which has become the center of the growing city.
Not even refugees would want to live near refugee camps
There's no principal like safety behind how restrictive many of these rules are, unless you count fear of people in different economic demographics to be a reasonable justification. It's a difference of opinion.
People who want more flexible zoning are growing in number, these are people who can't find an affordable place to live or are otherwise not happy with the changes made to zoning in the last 75-100 years, and they are driving changes in more and more localities.
OP is literally advocating for building refugee camps, not duplexes.