Smaller populations isn't a bad thing, but a rapid declining population is. At a stable birthrate of 2.1 there would be a workforce of approx 60% of the population compared to 20% elderly and 20% minors.
( Assuming productive age is 20-80 and all people die at 100 )
Once you enter a low birthrate of 1 that would mean at some point your workforce could be cut in half and your elderly population could be doubled. Now, 40% of the workforce will end up needing to sustain a 50% elderly population and 10% would be minors. If you're not carefull the loss of capital in the workforce could mean less people desire to have kids in a world that cannot sustain their needs. Leading to a feedback loop that is very hard to get out of.
One way of improving the situation is by cutting back on all non-essential programs for decades in order to stabilise the population at the magical 2.1 number.