Good point. What you're describing, at core, is leverage. The general principle of leverage is that it exaggerates the market's natural results. When things go up you stand to make a lot of money. But when things go down you stand to lose a lot of money.

One mistake (IMO) many people make about housing is not adequately thinking through the second possibility.