Well, you'd need both, not just one: landlord greed has a cap, average rents can't be much more than the cost of buying, though there will always be some folks who don't qualify for a mortgage for whatever reason. The only reason landlords can be greedy is the leverage they have because supply is constrained -- otherwise, their greed would be thwarted by developers' greed!
Speculation and landlord greed only really affect prices in a supply-constrained environment. In markets where housing is a commodity, and can actually be produced, the cost of housing tends towards the cost of land acquisition & construction. Not always smoothly, of course...if demand jumps very suddenly, the construction market needs time to adjust, etc.