Imagine the population is split in 2 subpopulations. 99% of the people want to have 1 kid per family, and 1% of the people want to have 3 kids per family. Initially the population will decrease quickly, but then it will start increasing quickly. After about 4 generations, the two subpopulations will be roughly equal (55% of the low fertility population, and 45% of the high fertility). After 9 generations, the relative ratio flips. At that point the total population is only about 40% of the starting one, but it's growing fast, because it's basically only high fertility individuals remaining. In 4 more generations it's above the starting level, and then in a few more it's 10 times the orginal population.

At the lowest though, the population is only 10% of the original population, and it appears to be imploding.