Yup. Unlike some countries like China, Japan does not go out of its way to catch dual citizens and it is extremely rare for somebody to actually involuntarily lose their Jp citizenship.
You can still get around this by going via a third country, but then they can ask for a return ticket from that country etc. The only foolproof way to evade the system and keep citizenship in a 2nd country is to acquire permanent residence in 3rd country!