Tokyo is certainly more dense than SV for the most part, but it isn't really
fantastically dense. It's very large, and has locations ranging from dense urban centers to mountains, but for the most part is rather less dense than you might think based on its reputation. E.g. a
lot of Tokyo housing stock is 1-2 story single-family dwellings (high-rise dwellings were pretty rare until fairly recently, in part due to the earthquake risk).
It has absolutely fantastic public transit everywhere, even outside the denser urban cores.
AFAICT, this comes from the way the city is structured: dwellings and businesses tend to cluster around rail lines and stations, increasing their efficiency, which makes it easier to support a large number of rail lines (rail transit is a profitable business in Tokyo), and that in turn increases the effectiveness of the network (the well-known "network effect"). Even though most rail transit is private and run by a huge number of different companies, they well-recognize the importance of the network effect, and so tend to cooperate, trying to make transfers easy and liberally using interlining.
There are also well-developed secondary transit networks (bus and bicycling) that increase the effective coverage of the rail lines.