You have the right perspective on this IMO. Thinking about owning a home as an investment is foolhardy: very few people are going to get rich off an increase in their home's value, and only a few are even going to significantly increase their retirement funds after downsizing.

Houses are places to live, not investments. It's in all of our interests if they're as affordable as possible everywhere instead of pushing the equivalence of an arms race.

The irony being, of course, that California is one of the few places where your home value could rise 5-10X and your property taxes appreciate at less than inflation.

Prop 13 creates many artificial barriers to selling property and other market distortions, and folks not downsizing when they'd like to is one of the crazier ones.