Which is perfectly fine! It still works, because houses do not have intrinsic price tags. They have intrinsic ordering of values, but their price is determined by what people are willing to pay.
Line up every person in the city in order of how much they're willing to spend on a home. Do the same with the homes available. Face the two lines across from each other. Everyone will buy a home nearby to what is across from them right now. Building a whole bunch more houses on the top end of the house scale doesn't leave those mansions empty, it just drives the mid-range homes further down the scale.
Supply shorter causes those further up the scale of "willing to spend on a home" to have only modest homes as options (what's across from them). At the same time, it pushes less and less desirable homes (often ones 2 hours commute from work) in front of those less able to afford homes.
If you want to argue that the divide between rich and poor is growing, and it's a problem, oh heck yeah I'm on board for that argument. But conflating the housing shortage problems with that allows the landlords to keep getting richer.