I think the real concern with this is not that we’ll have fewer people in Japan (a good thing) but that the system can’t support contraction.
If you have a workforce in their thirties who are still at entry level positions or thereabouts, and a majority of voters between the age of 50 and 80, you end up with a political class that wins elections with pro-retirement and healthcare policies and a tax-base that has no way to pay for it. This is a recipe for disaster and it’s already happening in Japan. It could easily happen in any aging population.