Sure, there is a housing marketplace, which can provide some correction. But houses are also homes. People are born in them. People die in them. They love them, they cherish them, they fight for them. There's a lot of truth in the saying, "Home is where the heart is."
What you're missing in that scenario is the large number of people who are displaced not just from their fungible, replaceable housing unit, but their home, their neighborhood, their family, and perhaps their city and their job.
Note also there are substantial class issues in the sentence, "Lower income renters move out, higher income families move in." The notion that poor people should take it on the chin because rich people covet their homes is not politically neutral. One could just as well say that it's the rich people, more able to cope, that should bear the burden of market fluctuations.
So I don't think, "is rent control even a good idea" is the right question. It's more like, "Are there more effective ways than rent control to reduce the harm?"