Immigration is not to replace, it is to substitute. So eventually Japan will be full of people again. Just of different culture and level of civilisation likely.
Even if it wasn't (like you seem to argue), I would rather realize than in 10 years many jobs will be automated, so you won't need as much manual labor and Japanese will be just fine.