Speculation isn't the problem. Speculation is like a time machine for prices. Speculators use information to discover the true price of an asset and keep buying until the difference between the market price and the "true price" shrinks to 0. In other words, they are making the market more efficient for buyers and sellers. Buyers benefit when the asset would be overpriced because of volatility. Sellers benefit when the asset would be undervalued because of volatility.

It turns out that the price of housing will go up in the future because of various reasons (primarily migration), but that information is available today. So what does the price time machine do? It transports future prices in the present and oh boy are those prices unbelievably high. The speculators are essentially whistleblowers. They are saying that the emperor has no clothes and promptly get burned on the stake.

Let's assume the opposite, that speculation itself was the problem, and not an external factor that is only brought to light. It would imply that the market price is exceeding the "true price" of the underlying asset and therefore the speculator will have a hard time selling his asset, unless another speculator buys the asset in an infinite loop. However, it doesn't change the fundamental problem, because at some point someone has to take the "true price" - market price loss because they are massively overpaying and cannot find a greater fool.

If you thought the latter was the case then you could just decide against buying a house because it is overpriced. Of course the problem is that no long lasting crash has occurred, even 2008 wasn't bad enough to keep house prices on a downward trend.