Balancing the population pyramid does not require number go up. It requires replacement level birth rates.

When that birth is far below replacement, you get a collapse that cannot be stopped once it gets going.