Housing cannot simultaneously be affordable and a good investment.
Homeowners on average tend to be more politically engaged, so homeowners dominate state and local politics. As for why every prosperous state doesn't have this problem as badly as California you can add prop 13 which hugely limited property tax increases for existing homeowners. This eliminated one of the major checks on real estate hyperinflation. You can further add the fact that the San Francisco Bay Area (the region with the highest prices by far) is geographically constrained and the fact that earthquake risk imposes additional site preparation and construction costs.
It's really a perfect storm for real estate price insanity, specially in the Bay Area. It's less insane in SoCal but still pretty high, especially when compared to median income in LA County and San Diego. SoCal is less geographically constrained.
There are also plenty of entrepreneurs and hackers and so on who move up here, so that balances things out to some degree. Still, though, California's NIMBYism has repercussions throughout the entire west coast.
It's almost as though people don't realize that lots of small things (like local zoning for example) add up to all the big differences that made the place so attractive that they moved there.
Specifically with regards to a bunch of retirees moving in, I'd take that over young couples any day of the week. People raising family will have much more impact than people who just want to enjoy retirement.
To me it certainly seems worse than other retirement approaches. You are forced into making a single asset a huge percentage of your net worth, which utterly destroys your diversification. Real estate is also notoriously illiquid, so if something forces you out of the area, you could take an enormous loss.
If someone proposed another retirement investment strategy with these characteristics that didn't involve your home, people would say, "I'll pass... that sounds like a really bad deal."
Let's say you buy a house at price 100, and the market doubles bringing the price up to 200. If you then sell and move into a house that was 120 and is now 240, then you've got to find 40 extra for the upgrade instead of the 20 extra you would have paid absent the price change.
That might hurt in the short-term, but in absolute terms your asset is worth a lot more, so you're winning. And depending on how much of your mortgage is paid off, you could be ahead in the short-term too; if you have paid down 25, when you sell you get 50 back to put into your new mortgage, and that more than covers the increase in cost of the upgrade (you got 25 profit from the market movement, and had to pay 20 extra for your upgrade, so you're up 5).
Not to mention that if you don't decide to move house, the nest egg that you're leaving to your kids (or sitting on until you downsize later in life) just doubled in value.
As you can probably see it would be easy to construct a set of parameters where the short-term price change hurts you (e.g. you're moving before paying down much of your principal), but property owners tend to be optimizing for the long-term gains.
Yes, lots of people plan to sell and move to a less expensive locale in retirement. Often, people who came to expensive places for good jobs but not because they had any particular desire to live in that place plan to return to where they came from in retirement. I've known lots of people from the Midwest working in CA where this is their plan.
Not always. Being able to flip a single family to a developer who is going to build multifamily is often the best return on your investment. The more confidence the developer has that their proposed building will be approved, the higher the price you can get for your lot.
You profit from this, but at the expense of your neighbors. Imagine the opposite scenario where all over your neighbors do this, but you're the single hold out. Would your home be worth more, less, or the same with twice (or more) the number of people living on the block? Obviously, we don't really know, but the holdout is likely fearful that their property value would go down dramatically.
You can see why people who have $1,000,000 mortgages don't want to see the value of their house fall to reasonable levels, because they'd be the ones taking the six-figure bath.
As a single family house its value would be less. But since all your neighbours already flipped their home for a developer building multifamily buildings, you surely can do the same, and as land to build an apartment building, your property probably is more valuable than it ever could have been as a single family home.
Not if you own the underlying land, or at least a portion thereof, and willing to build higher and denser and rent it out. For example, if I own a 1 story house, NIMBYism prevents me from building another story and renting it out as a separate unit for extra income. NIMBYism raises the value of my current property but lowers its potential value.
In the long-run you want economic development to happen though.
Edit: I don’t retract my statement, but at the time of posting this my parent comment only contained the first paragraph. I agree with most points brought up
I also think closet classism/racism plays into it. California has a liberal facade but there's a lot of "we don't want those kinds of people" in our neighborhood sentiment below the surface. Making sure housing is unaffordable for minorities and the poor is a great passive-aggressive way to implement segregation without having to admit it.