Who is swimming to Japan? Joking of course, but you have to consider that unless there are land adjacencies, it’s unlikely invasion is a concern as population declines.

Less Japanese means more open space and less land contention from remaining Japanese. Less people overall means less resource contention overall.