Thank you-
With respect, "government regulations" doesn't begin to capture the dynamics, and IMO is not the right way to understand the ecology.
An assumption underlying supply/demand from an economic and game theory sense is win/win- producers want to produce, consumers want to consume.
Supply/demand- or really any- mechanism design breaks down when there are vast discrepancies in "power" between the actors, when some actors can change the levers on the mechanism, when actors have substantial incentives for sabotage, when the interests of the actors are simply in opposition.
In real estate, owners and dwellers are, to a much greater degree, entities in opposition to one another, and only to a small degree, united in a common interest.
Just one example- much of NYC's population is elderly, on a fixed/unindexed income. They simply cannot participate in a dynamically priced housing regime. There is an extensive history of owners preying on this population. If one reads stories of old New York, one sees that the vignette in The Godfather II where the young Vito Corleone strongarms an owner into keeping on an elderly tenant was extremely well chosen.
Similarly, on a farm, children used to be a highly economic investment. In a city, children have become about the most uneconomic investment a family can make, to a degree because of de facto cartel behavior on the part of owners, where pricing per sq foot differs substantially by bedroom count.
The presence of elderly and children must be optimized for for a city to exist at all, as the income producing working adults have both in their vicinity. From a pricing perspective it used to be that rent control and rent stabilization kept prices low for certain segments of the population. Stable/low prices are, to put it mildly, not a desirable opportunity for owners.
This game was remachined with things like section 8 housing vouchers. Rather than prices being stable, they can "float" with the "market" (I use the term "market" loosely as there are a rich spectrum of techniques to shift what is considered market pricing). Tenants who pass through a gate are limited in what they have to pay to 30% of their income. The gap between 30% of income and "market price" is paid to owners by the federal government.
And so on. Much more could be- and has been, of course, by people much better suited than me- on evictions, or money laundering, or any number of other subplots.
Real estate is a balancing act, by governments, between the needs of what amount to predator/prey populations.
In regards to the latter point- doubts about widespread cases of vacant properties being tolerable/desirable- forgive me, you evidently have not been here. Just one point in time snapshot for one segment of the city:
http://www.vacantnewyork.com/
Pervasive vacancy is a malignancy, a cancer, and for lifelong NYC residents- prior to COVID- was a source of deep personal sorrow. The system was broken, and not in any way that supply/demand offered any wisdom to fix.
As I suggested above, people who have the hammer of supply/demand often want to treat housing like a nail. It is not. There are lots of rich and intricate dynamics, and the language of supply/demand does not begin to be able to account for either the data, or the stories. Hope that's helpful.
Cheers.