If you don't live in Japan, use your US passport to enter/exit Japan and only carry that passport with you. They might be able to see that you also have Japanese citizenship, but they won't be able to take your Japanese passport if you don't have it.
If you fly from the US to Japan and show a Japanese passport, your chances of getting caught are much higher - all they have to do is ask why you don't have a US visa stamp in your Japanese passport.