> the people who are negatively affected by this are people who have already decided to move.
"Decided to move" is not a reasonable description of the reasons for people to move into cities - they want jobs, they want to live among their peers, they want to take their place in the world. They didn't "decide to move", as if somehow they could live in their hometown all their life achieving their dreams.
> the people who are negatively affected by removing rent control will be people who don't want to move but are forced to move by rising rent.
I want a pony. And a boat. But I don't get those things, and I'm certainly not going to demand that I get to buy those things at a price far-below market rate. It might be nice to allow a group of people to not have to move "because they don't want to" rather than because they can't afford to stay - but it's unjustifiable when it comes at the expense of locking an entire generation out of the rental market and locking out younger people from living perfectly reasonable lives.
You can't defend a privilege that seeks to entitle one group to live according to their whims, isolated from the economic realities in which everyone else is forced to live. Rent control is a zero-sum game, it only provides benefits to one group by screwing over another.
> it seems that people having to leave their neighborhoods because of rising rent is a larger negative impact (both to them personally and to society as a whole) than people who are already moving having to pick a neighborhood that is not their preferred neighborhood.
You're trivialiaing the problem. This isn't about satisfying people's whims or preferences. An entire generation is being locked-out of living where they work, with many being shut out of jobs because of that. The person who can't afford to move to SF doesn't live in a less desirable neighbourhood of SF, because they can't afford to live there either, and they don't move to Oakland anymore, because that's expensive too now (guess what, they have rent control too). So is Richmond, and Almeda. So they move where? Concord? The commute alone is going to eliminate them from many potential jobs in SF. Meanwhile you defend single-earner part-time worker older couples living on Nob Hill (and who likely have ample savings) working irrelevant jobs which could be anywhere because they "don't want to move". One person's whim does not outweigh another's necessity.
Never mind the issue that rent control is probably harming the very people it aims to protect by putting so much pressure on the rental market that they can never afford to move, and so much knock-on pressure on the housing market that they can never afford to buy. So even if you're convinced of its social value, it's still non-sensical.
A fair market and a livable minimum wage would seem like an infinitely better solution.