Clearly, they worry about China, but they really worry about how the Japanese military answered to no one in the last decade of Imperial Japan.
This is what concerns a lot of Japanese people. The military wrote a check that the Japanese people had to cash.
Massive populations and political revolution are the common cause. Japan doesn't have the conditions for either, even if Abe achieved his relatively anodyne constitutional amendments. China is more susceptible to adopting that expansionary role, and it doesn't have a constitution binding its hands.
Japan uses Chinese characters not because they were ever invaded by a Chinese army. Soft power isn't a modern American invention. The Chinese mastered it a thousand years ago!
That's a narrative story, not a history.
"China" has never been a continuously consolidated political entity. The PRC (and the ROC, and previous imperial governments) find it a useful myth to equate its current geography of control with a tradition that hearkens back through millenia, but it's quite simply not. The Qin dynasty initially controlled the area of a small province, and conquered its neighboring state rivals one by one to establish a dominion over them; it fell, just as countless other empires fell, leading to centuries of alternating fragmentation and consolidation.
Even in modern times, various political entities that position themselves in the history of China have engaged in expansionist colonial missions over a range of territories, from Xinjiang to Taiwan to Tibet. Even Inner Manchuria is only considered "part of China" because the Qing dynasty ruled it and China simultaneously. (As an interesting aside, contemporaneous Han historiography considered the Qing ruling class foreign usurpers; it was only in the 20th century that ROC/PRC historiography decided to incorporate the Qing Dynasty into China's lineage, for political reasons.)
But I also know quite a few Japanese people in their early thirties, some of them very very very to the right of the political spectrum. Even to them Imperial Japan is sheer terror. Young kids being forced to commit suicide (kamikaze), endless suffering of the population, a crazy mentality, worshiping the emperor as a god, horrific experiments in the Chinese countryside, etc. My sample might be small, but I don't think there's much nostalgia of Imperial Japan anywhere in the Japanese population.
[1] Extra points if you can't even recognize Abe doing giving an interview on TV. And yes, that one has never missed an election. That's the most terrifying part to me. If you want to opt out of caring about politics, it's completely fine by me, but at least do it completely...
You have this idea that the Japanese military will become a runaway juggernaut subjugating Japan under its will, and this is simply ridiculous. No signs point to it.
I could almost accuse you of fear mongering
Also, your accusation that most Japanese are "very uneducated when it comes to politics/history" is anecdotal and incomplete at best, comically reductive at worst. Do you even read Japanese? Do you not see the amount of considered, deep analysis and discussion on TV and in the newspapers? Do you not see the huge numbers of books on history in the bookshops?
In any case, it is ridiculous to claim that "most" (or even some) Japanese people live in fear of the SDF. You'd be laughed out of the room for claiming that in front of anyone with even a passing knowledge of Japan.
No one in Japan is afraid of the Japanese military.
Utter fiction