I wouldn't say it's the vast majority if Tokyo/Chiba/Saitama/Yokohama holds like a third of the population. Add in those other cities you mentioned and the majority of the population lives in cities.
Of course in any developed country the majority of the population lives in cities. What I mean is that Japanese cities, outside of a few core areas, are really ugly and depressing.