Homeowners in areas where housing is totally unaffordable area either extremely wealthy, by definition, because of their property ownership, or they're wealthy enough to pay a mortgage on an absurdly expensive property.
Where housing-as-good-investment is successful is really only in areas where shortages exist. You're not going to harm the middle class folks in Tracy by fixing the housing crisis in San Francisco.