Construction costs aren't much higher than they have been historically. Slightly higher because of demand for things with inelastic supplies (like tower cranes), but we're talking average cost change by a few percent at most.

And yes they would keep building. Land owners are the ones who gain/lose by fluctuations in demand, not developers. If demand drops and rental prices drop, land owners lower their prices that they sell to developers. This has been proven so substantially by Housing Economists that it's basically considered consensus by now. Spend some time with the Journal of Housing Economics if you would like to learn more.

The real reason why developers only build luxury buildings is that they can't build enough to satisfy all demand, so the few buildings that they can build will obviously go to the types with the highest margins. If we allowed for more development, developers would still build luxury buildings as long as there was unsatiated demand for them, but they'd also build normal and affordable housing as well.

But developers ARENT allowed to build. That's the problem.

Yes, let's get rid of all these building height limits so that builders can actually build. That would be wonderful.

NYC really is the perfect comparable to see the potential future of SF. NYC has a housing glut, but a it's all luxury housing. No developer will ever build new "affordable housing" because the economics don't make sense. Increasing the housing supply should decrease prices in theory, but in reality more people will just show up and pay the current asking price (or more). Price per sqft in NYC is pushing 2-3k+ while SF is typically around 1200 per sqft in the nicer neighborhoods like Noe. Pacific Heights is still the priciest at around 1500-1600 per sqft, but still nothing compared to a lot of NYC. This is my clearly my own opinion and it likely won't be popular, but I'm a little surprised that SF is still as "affordable" as it currently is, and it's my view that it will never again be priced any lower (except for short term trends such as recessions).

But... Why are those people moving in? Is it perhaps possibly because these people LIKE to live in high density spaces?

If people keep moving in that means that high density is by definition desirable. We as a society should be building the stuff that people want, and your argument proves that what people want is high density living.

The fact that high density living attracts new buyers is even MORE reason to make more of the type of living that people love.