It's not for poor people (no doubt) and it's not really for billionaires either, for people so untethered from jobs and material needs that place is irrelevant.
If you can pick up and pay cash for a Tahoe house tomorrow, sure, you might leave. But that resident was never very plentiful. They might make for an interesting anecdote, but in the big picture, they don't matter.
SF's core value, today, is for this person:
You work for Google, or Apple, for FaceBook, or an accomplished startup. You are a senior engineer or an executive. You may have gone to an Ivy League. You want to live in a fun, world-class, brand-name city - but not Palo Alto or San Jose. Where can you go?
San Francisco is pretty much alone in its category here, as the type of city that type of person likes. There's no good substitute in the area; "a really clean Starbucks and a top-notch Target" ain't it, and will never be it.
While there may be fluctuations in house sales and so forth, I think SF will continue to rank highly with that demographic. And as long as that's true, housing prices will be safe.