People of Japanese ancestry top out at around 1% of all Brazilians. That’s not a lot or significant portion of the population. If you get to 5 or 7% we can talk about significance. On the other hand they tend to have outsized influence on the country, so you may be projecting that onto pop size.
Brazil is a gigantic country, though. 1% of 220 million is 2.2 million which is around 2% of Japan total population.