Ah yes, having both probably makes the prioritization fall apart :(

I have no idea if there's any sane way to make this work. Since I only speak Japanese, my fix on Linux is to prepend IPA PGothic/IPA PMincho (which only contains Japanese glyph) to lang=ja, which worked quite well.