Your analogy is logically sound, but it doesnt match what I've seen in reality. I think the extra factor is that rich people seem to attract more rich people. That is the number that I think can't be out built. The rise in prices attracts more of them. Or rather, the fewer poor people around tends to mean rich people find it even more desirable. In practice, no government or real estate boom can out build this network effect of rich people attracting more rich people who eventually decide they like things the way they are. Not just real estate, but everything about the place. And they now own it, so its not that crazy to imagine they get to decide this. Im not morally in support of it, just stating history.
Rich people are especially into "knowing the best", whether its vacations, cars, and yes, places to live. Thats what I meant about them wanting to come buy stuff in your town even if they don't currently live there. And the data kind of shows this. California has an outflow of poor and and inflow of richer.
Basically, desirability is inelastic. California has better weather than a lot of places. The only people who can pay up the vertical inelastic curve are rich people. There are a bunch of them who want to move to your town from elsewhere.