There are like 50K Indians in Japan, mostly Tokyo. This is a surprisingly low number. Sure there is the language barrier but that barrier holds true for Germany also which has a lot more Indians.
My hunch is education - Japan does not have too many universities to attract young talent, as compared to Germany, Australia or the UK. I'm not sure if this is by design or if there are some other factors at play.
If Japan built a lot more universities and made it easier and relatively affordable for foreign students to sign up with English classes etc, perhaps there will be a lot more immigrants from countries such as India, Vietnam, Cambodia etc.