I think housing in SF is very dependent on 1) the tech boom and 2) the baby boomers who own property in the south/east bay (but wont sell because they missed out on the 2008 sales price). Both of which are temporal in nature.
SF may always be a enclave of the wealthy, but if the rest of the bay area goes to shit, prices will drop. It may not get massacred like SJ eventually will (after the baby boomers houses start hitting the market en masse), but even those wealthy enough to live there will feel it.