I take a very SFBA perspective here, but...
> Without it they would have been displaced even earlier
This is mostly hypothetical. In functioning housing markets that don't manipulate prices or limit new construction through policy, speculative bubbles make little sense -- why would housing become more expensive if it's easy to build lots of it?
The argument that removing just rent control would make folks worse off -- that I'm sympathetic to, of course. Obviously you also need to build. But rent control as practiced in most of the Bay Area causes lots of grief despite only helping a relative few of the desired beneficiaries.
That is, unless you define beneficiaries much more widely than you do -- to basically make it a benefit to seniority. Rent control, like Prop 13 and construction restrictions, work to allocate scarce resources based on seniority rather than wealth or ability/willingness to pay. The argument that it allocates based on need (i.e., poverty, inability to move house) is weak; that's a mild side effect, and policies that just straight up subsidize housing (i.e., vouchers, etc.) for those in need could be much better targeted, and could be funded in a way that doesn't also incentivize poor maintenance, harassment, and eviction.
Is allocating based on seniority better than wealth? For current residents--at least, until they're Ellis'd--sure. For their kids, who lack seniority? Almost never. Everyone needs a place to live. Right now, more people want to live here than we're allowing to (through construction limitations). So, sure, we could argue about how to allocate -- or we could just end the damn scarcity!