Exactly that, it can very quickly snowball and then you just don't need as many buildings anymore.

Or look at it this way: https://www.pewresearch.org/hispanic/2020/08/20/facts-on-u-s...

> There were a record 44.8 million immigrants living in the U.S. in 2018, making up 13.7% of the nation’s population.

If the USA was net neutral on immigration (as many people entered as left), the birth rate would be too low, there would be some tens of millions of dwelling units either empty or never having been built.