Americans in particular often believe that Japanese attitudes are morally wrong. And it's not so much bullying as it is that Japanese cultural stances are immovable. You can throw yourself against that wall as much as you like and it is not going to budge. So if you honestly believe that some things are morally wrong and you call people out on it... well you are going to grind yourself into dust.
Although I shouldn't get too carried away, I'll tell one last story. When I worked at the high school here, my supervisor was trying to get me to work without compensation on weekend (there was a special event). All of the other teachers got a day off in lieu, but my supervisor didn't want to upset the apple cart by trying to get the same treatment for me (I was on contract, so played by different rules). This is pretty typical, and I've seen other foreigners absolutely get squashed by the system as they try to preach about worker's rights.
Instead, we had a drinking party that week. So I sat down with the department head and a bottle of nihonshu (Japanese sake) and we preceded to enjoy ourselves. At the appropriate time (just before he passed out), I casually asked him if the regular teachers were getting time off in lieu. He said, "Of course". I laughed and said something about the unfortunate lives of those who work on contract. He stood up (unstably) and assured me, "No! It's not right. We can fix this!". At which point he yelled over to the principal on the other side of the room, "We can get a day of in lieu for Mike-kun here, right? He's a good guy!" To which the principal laughed and said, "Of course, of course!".
My supervisor was clearly livid the next day as he brought the schedule and asked me when I wanted to have my day off. This is Japan.
Foreigners often have difficulty with this because rules in most western cultures are negotiable. You can appeal to common sense. You can argue your position and modify your environment. This is generally not possible in Japan. The idea of drawing a line, as you have described, will almost certain end in an extremely difficult position if you wish to be inside.
I think there are 2 happy places in Japan. You can decide that you aren't going to follow the rules and accept to be outside. Everyone will smile and be polite. They will be very helpful to you. They will praise you on being able to use chopsticks, etc, etc. You will forever be an honoured guest.
But not many people want to be treated as a guest in their own home, so most people want to be inside. This is also fine, but you have to learn the crazy rules and follow them exactly. You have to be OK with being reprimanded for making the slightest error. You have to slot yourself into society exactly where they put you. You have to do what is expected of you.
Where you get into trouble is when you want to be inside, but you want to pick and choose which rules that you wish to follow. Or if you wish to negotiate those rules. Or if you wish to convince others to play by other rules. That's just not going to work in Japan and I think it's why most foreigners don't stay, even though it is technically very, very easy to immigrate here. Like I said, especially if you have a very strong moral sense of right and wrong, it's going to be really, really difficult unless it happens to match exactly the sense of right and wrong in Japan (which is pretty unlikely if you were not raised in Japan).