On top of that in Tokyo there are 23 wards each with it's own laws, taxes, etc.
What's special about California? Why does it work in Tokyo with 25 million people (or maybe it doesn't?) and why doesn't it work in the bay area with only 9 million?
Much of California public services are provided by local taxes. I'm not sure about Japan, but if it's anything like Canada then much of it provided at the state or federal level. Add in things like Prop 13, "Pay for government?! That's Somebody Else's Problem", and you end up with huge incentives to externalize your costs.
Schools are cheap if all your workers live somewhere else. Low-income assistance is easy if the cheapest place to live is $2000/month for a room-share. Most of the local legislators are home owners and decreasing costs by increasing supply will directly hurt them and their paper gains.
If one local wards would take decisions to enrich itself to the cost of others the system would break down. It wouldn't last a day in California.
That said, there are some problems there too, such as the aforementioned quintillion public transport lines. (Source: a few friends who moved there)
Japanese zoning and urban development policy are set by the central national government. That policy is considered the gold standard for simplicity and promoting quality development by planners everywhere in the developed world.
If the quality and honesty of government urban planning that Japan has could be transplanted, life in the Bay Area would improve immensely. No amount of land scarcity or Google shuttles or rising demand or offshore Chinese corrupt officials parking money or anything else would be able to make things unaffordable if local governments were good, honest, and effective. But quality government is simply not possible with the current politics of California.
[0] http://urbankchoze.blogspot.com/2014/04/japanese-zoning.html
Nnnno, they're not. In fact, Japanese zoning -- which, incidentally, is set at the federal, prefectural and local level -- is widely regarded as batshit insane. For example, there's a giant garbage incinerator right in on the northern side of central Tokyo (Ikebukuro); taxation makes "used" houses nearly worthless and thus encourages building new cheap crap; draconian building restrictions make Japanese apartments tiny; the well-intentioned "sunshine laws" have all sorts of perverse side effects like making it well-nigh impossible to build tall, high-quality residential buildings in the city center but encourage tall but cheap, crappy ones in the middle of nowhere, etc etc.
This Nomura Research report is an excellent read: http://www.nri.com/global/opinion/papers/2008/pdf/np2008137....