NYC really is the perfect comparable to see the potential future of SF. NYC has a housing glut, but a it's all luxury housing. No developer will ever build new "affordable housing" because the economics don't make sense. Increasing the housing supply should decrease prices in theory, but in reality more people will just show up and pay the current asking price (or more). Price per sqft in NYC is pushing 2-3k+ while SF is typically around 1200 per sqft in the nicer neighborhoods like Noe. Pacific Heights is still the priciest at around 1500-1600 per sqft, but still nothing compared to a lot of NYC. This is my clearly my own opinion and it likely won't be popular, but I'm a little surprised that SF is still as "affordable" as it currently is, and it's my view that it will never again be priced any lower (except for short term trends such as recessions).

But... Why are those people moving in? Is it perhaps possibly because these people LIKE to live in high density spaces?

If people keep moving in that means that high density is by definition desirable. We as a society should be building the stuff that people want, and your argument proves that what people want is high density living.

The fact that high density living attracts new buyers is even MORE reason to make more of the type of living that people love.