Not sure what the scam is. Sounds like normal supply and demand.
https://en.wikipedia.org/wiki/Jevons_paradox is probably the answer: Non-idiotic land use would make the cost of floor area decrease, but the cost of land increase.
If, instead, we hadn't massively down zoned everything in the 20th century, an upzoning wouldn't be "giving away" lots of value, because there would be an abundance of locations that could be used.
This is why small focused upzonings like those that happen in NYC and SF are so disastrous and destroy neighborhoods. Instead of gradual city-wise change like what happens in more affordable places like Tokyo, SF and NYC are super expensive because they set zoning so almost everything is "built out." Any area that is "built out" has reached complete failure in city planning, in my opinion. Which is pretty much all of the Bay Area.
But housing prices are affordable in comparison to places that have restricted use in less efficient ways.
"Higher" is a relative term, and the question of what the baseline should be is quite complicated. They are also quite variable depending on what you can do with the land; if use is extremely restricted it can keep a plot of land down in price, even while it inflated the cost of land overall in the market.
It’s an incentive problem, and a lot of people are incentivized to keep supply down.
Housing can be affordable or it can be a long term investment, but it can’t really be both.
If a lot of people are desperate to have their property grow in value, they will fight anything that threatens that.
I think it’d be interesting to revoke prop 13 in a staged way. You can either build some large X amount of new housing each year or you can accept a 10% increase of the property tax on your property each year until it reaches the market value rate.
If you’re going to constrain supply you should have to pay the cost.
Though for the same power reasons this could never pass politically.