"I'm sure there are cases where this isn't true, but as a general rule, any opposition to new housing in SF concedes ground to wealthy owners at the expense of working-class renters."You're missing a few critically important pieces of context:
1) The city of SF gives developers two options for the low-income housing set-aside -- they can make a percentage of their units low income, or they can contribute a percentage of the project cost to a city-wide fund.
2) Developers hate both of these options, but they far prefer the latter one (donate to a fund), since it's a fixed cost.
3) Because the city is in competition with these same private developers for land, the low-income housing fund is at a significant disadvantage when it comes to actually building low-income housing. IIRC, the city has only built a few projects in the last decade.
4) As a result of 1-3, most of the community resistance to new construction ends up being about forcing developers to choose the first option (i.e. actually building low-income units). They use the only political leverage they have to compel this choice: the planning process.
5) Many/most skirmishes about permitting get characterized as "NIMBY opposition", when in reality, they're about low-income housing set-asides. It's just easy to point to the zero-influence yahoos who show up to any public meeting, and mischaracterize them as the source of the problem, so developers do this.
Basically what I'm saying is: unless compelled, there's no situation in SF where developers build anything other than luxury units. There's no ROI otherwise; land is too expensive. And they won't build new luxury units into a down market, either, for the same reasons. Low-income housing requirements are the only way this stuff happens.
So I'll turn your question around on you: name one US market that has successfully lowered real estate prices via construction. Even in places canonically held up as examples (e.g. Houston), affordability is maintained by sprawl...which is not an option for San Francisco. For the exact economic reasons you've cited, nobody buys or builds real estate that they know will lose value. That's the fundamental problem with the "build out of it" argument.