Americans in particular often believe that Japanese attitudes are morally wrong. And it's not so much bullying as it is that Japanese cultural stances are immovable. You can throw yourself against that wall as much as you like and it is not going to budge. So if you honestly believe that some things are morally wrong and you call people out on it... well you are going to grind yourself into dust.
Although I shouldn't get too carried away, I'll tell one last story. When I worked at the high school here, my supervisor was trying to get me to work without compensation on weekend (there was a special event). All of the other teachers got a day off in lieu, but my supervisor didn't want to upset the apple cart by trying to get the same treatment for me (I was on contract, so played by different rules). This is pretty typical, and I've seen other foreigners absolutely get squashed by the system as they try to preach about worker's rights.
Instead, we had a drinking party that week. So I sat down with the department head and a bottle of nihonshu (Japanese sake) and we preceded to enjoy ourselves. At the appropriate time (just before he passed out), I casually asked him if the regular teachers were getting time off in lieu. He said, "Of course". I laughed and said something about the unfortunate lives of those who work on contract. He stood up (unstably) and assured me, "No! It's not right. We can fix this!". At which point he yelled over to the principal on the other side of the room, "We can get a day of in lieu for Mike-kun here, right? He's a good guy!" To which the principal laughed and said, "Of course, of course!".
My supervisor was clearly livid the next day as he brought the schedule and asked me when I wanted to have my day off. This is Japan.