I mean, I'll be honest and share that I didn't know the housing regulations are so f-ed up in the US, but they are pretty complicated everywhere.
What drives housing prices is a deflation of living goods coupled with an inflation of luxury goods which makes finding safe investments for large sums of money with relatively high yields hard to find. This is driven by an increasing inequality gap, which makes rich people desperately trying to find more things to invest their money into. And they have found housing in urban areas is the safest bet right now. So houses are bought and sold for the millions in hopes somebody else buys it for higher prices or the area justifies the rent.
Sure, but that part immediately follows - there was a good piece from a property developer working in Los Angeles (which has a lot of those kind of regulations) that was posted here a couple of months back. Because of all these regulations (parking minimums, max heights, setback requirements and so on), the only kind of housing that is profitable or even possible to build is luxury housing - the land costs x, the things that the regulations require cost y, you have to charge a lot more than the city's existing housing stock (most of which is grandfathered in) so you can only justify that by targeting the high end.
There is definitely speculation in the housing market, but most or all of the price rise can be explained by simple supply and demand. And since housing demand is fairly inelastic (everyone needs a house), a small amount of undersupply means a big increase in price.