As Canada has grown rich, our fertility rate has gone down. I have no moral problem with that, but it does cause a problem: as the population ages, the ratio of workers (who pay tax) to retired people (who carry a tax burden) shrinks. The options are increasing the taxes on workers, cutting the benefits to the retired, or increasing the working age population via immigration.
As to grocery stores, housing prices, etc. I think these problems are more related to a lack of competition. A free market can handle an influx of consumers.