> Young people just don't want to live in them.

That's why the hometown tax exists. Small towns use their residence tax money to raise children then none goes back in when the young adult moves to Tokyo and pays residence tax there instead of in their hometown.