You don't need a considerable increase in population to affect prices substantially. You only need to push out the marginal people, the ones who are at the lowest end of the rental spectrum. So to compete, these people are forced to pay a higher % of their salary towards rent, this creates a vicious uptrend in real estate prices.

The only remedy to this problem is to increase housing supply.

Good luck...