To clarify, by "directly fighting against" I meant "there were battles in which Japanese forces actually engaged [other country]'s forces" rather than just "Japan was in one big alliance and [other country] was in an opposed big alliance." That's the specific set of countries Japan holds higher levels of lingering xenophobia toward.
You're right, in that the people holding grudges today aren't likely to have directly experienced these events, but more likely are those who saw their fathers/brothers shipped to the front and never return, and so who hold a vague-but-strong grudge against whatever nebulous enemy the Japanese propaganda machine represented as being on the other side of those particular battles.
(Sadly, this can be a stronger foundation for xenophobia than actually having fought an enemy corps-a-corps, as these folks have never actually been exposed to the experience of seeing enemy soldiers hurting/dying, grieving, wishing they weren't at the front, and just generally being human.)