NYC is often considered more expensive, and Manhattan is certainly... well... Manhattan. But taken collectively the NYC metro area is actually much more affordable. You can find places to live that are nice, friendly to someone older than 25-30, and at least sanely priced. They might not be in the closest or most fashionable neighborhoods, but they'll be reachable by NYC's excellent transit system.
In SF you have nowhere good to go unless you want an absurd commute, and its transit is horrible compared to NYC. So you're either rich or you're in a crummy apartment or with roommates. That doesn't work for people who aren't kids in their 20s.
My suspicion is that it's part policy choice -- a lot of protectionism from existing property owners who want to keep riding the property bubble and a lot of NIMBYism -- and part irresponsibility on the part of homebuyers. I suspect that many homebuyers in SF are doing the irresponsible thing of squeezing into a home by spending upwards of 40%, even 50+% of their income on mortgage payments instead of the usually fiscally responsible number of ~30%. And this is punishing everyone and sucking the soul from the city.
In the long term, Silicon Valley and SF is eating its young. It's a city built on young entrepreneurs coming there and doing new stuff, but if nobody who hasn't already made it can afford to live there what happens then? The other problem is that these high costs soak up everyone's spare time and money, leaving them with the requirement to work like crazy for other people to afford to live instead of on their own innovative projects. It sucks away the marginal "play time" when innovation really happens.
If this isn't remedied, SF/SV will become an echo chamber of already-made-its and already-know-it-alls. Then it will stagnate and be left behind.
Edit: real estate hyperinflation in general is a social ill that deserves to be tackled directly as such. It seems as if real estate has mutated into this hungry sponge that sucks up all surplus economic vitality from any region that experiences success. Imagine what all that money in SV could be doing if it weren't all sucked up by real estate.
It's like beyond a certain point, people no longer get wealthier. Their houses do. The home -- the "American dream" -- has become a tool of indentured servitude and bank fiefdom.