These "wealthy residents" are poorer in their hometowns. They don't migrate out of fun. In middle of nowhere they may get 40k per year and in the bay area it may be 160k or more.
Prices are set by the balance of demand and supply. The demand isn't going away. The supply is what we can do something about.
Even building expensive luxury apartments helps, unless no one can afford them at all (which is not what the builder wants to happen). The people who rent those aren't taking up spaces others could have afforded.