Nice theory, but my experience is exactly the other way around.

Even after several years of learning Chinese I still had trouble communicating with Chinese people, especially those who had no experience talking to foreigners. When I arrived in China and asked the way to the university I was going to (which was close by and very famous) they just didn't get what I was saying. In the end I had to show them the written word.

I don't speak Japanese, but when I arrived and said the name of the city and they immediately understood where I wanted to go. After my experience with Chinese I was flabbergasted that that went so smooth.

I blame the tones in Chinese (which I admit I'm not very good at)