I suspect the proximate cause of the drop is the hysteria about overpopulation that gripped all Asian countries in the 1960-1990s. The governments there raised two generations of people socialized to believe that having too many kids was bad for the country.
Japan and Korea never had anything as barbaric as China’s one-child policy, but the government heavily propagandized population control during the second half of the 20th century. They imposed taxes on families with more than two kids, heavily promoted abortion and sterilization, etc.
We've disguised it with massive immigration.
There may come a time when a new way of thinking will spread, causing this trend to reverse.
>The UN forecasts that Japan’s total population could plummet to 104.9 million by 2050, possibly even as low as 87 million by 2060;