It's truly saddening that such a stance can still work when it's likely the average citizen will not encounter an immigrant in their day to day life. a million immigrants is not threatening the jobs of 300m Japanese people.
Tourists are short-term visitors who are there exclusively to spend their money in Japan and leave it with its citizens. If the Japanese do not want that because the tourists don't come fully prepared for living in Japan, then you should just deny tourist entries to the country. It would be win-win for everyone, because there are plenty of other countries who would gladly take those tourists instead.
You are correct that we do not threaten jobs either. A large majority of the foreign population is working low/unskilled jobs. Generally, the native population is not wanting those jobs.
They get blamed for a lot of crime; idk if that's true or not but it probably is, in part because American culture has less respect for authority, in part because American culture has more respect for individual liberties, and in part because any time you have a large enclave of foreigners (regardless of where they come from or which host nation they're in) they always end up committing more crime than the native population. They also get blamed for driving up prices in the real-estate market (this is definitely true, the US Navy owns 20% of Okinawa).
Blaming "immigrants" instead of specifically blaming the US military is also very convenient for both the US and Japanese government because both governments are largely in-favor of continuing the status quo so it's not surprising that politicians would obfuscate the source of the problem by blaming immigrants as a whole.