This does not happen, nor is it remotely desirable. Having a life involves much more than having a decent job. It entails living where the person chooses to live according to their personal tastes and desires.
If you force the decision on where to live based on a single constraint that in theory is only a major part of your life so that you can fund your personal life and personal interests then this leads to poor quality of living for all (including spouces, offsprings and immediatr family) just to cater to a company's bottom line.
You're confounding your observations. People with money choose to live in areas that maximize their quality of life, and they don't mind paying a premium for that.
Quality of life matters to all, whether rich or poor. Poor people aren't masochists.
The main driver of price is not "demand" imagined as how much people willing to pay for a particular house but demand as how much people are capable of paying, which is mediated by access to debt. If access to debt is clamed down on across the board, then prices will fall accross the board. People will still compete on price but as a society a smaller fraction of income will be spent on housing.
it's not the whole story however. In order to increase Supply, it has to be profitable to build new homes. If building new homes is not profitable, none will be built, and the price of existing homes will be forced upward until it matches the cost of new construction.
So I agree with you that access to debt is a very important factor, but I disagree that it is more important than demand when you look at the whole picture.