I realized I answered the question (if landlords/investors restrict housing supply) quite indirectly, while there is a more direct answer.

I recommend Keen/Standish paper on the theory of the firm: https://www.paecon.net/PAEReview/issue53/KeenStandish53.pdf

They show that profit-maximizing agents communicating via price-setting only will happily restrict output in order to reach oligopoly prices.