Housing cannot simultaneously be affordable and a good investment.
Homeowners on average tend to be more politically engaged, so homeowners dominate state and local politics. As for why every prosperous state doesn't have this problem as badly as California you can add prop 13 which hugely limited property tax increases for existing homeowners. This eliminated one of the major checks on real estate hyperinflation. You can further add the fact that the San Francisco Bay Area (the region with the highest prices by far) is geographically constrained and the fact that earthquake risk imposes additional site preparation and construction costs.
It's really a perfect storm for real estate price insanity, specially in the Bay Area. It's less insane in SoCal but still pretty high, especially when compared to median income in LA County and San Diego. SoCal is less geographically constrained.