> Why does this happen in Japan but not NYC or SF?
I don't think zoning is the complete answer here -- there's a lot of stuff going on in Japan, and in Tokyo in particular, that's very different from the US.
For example, a lot of cheaper places are tiny and have no shower or a shared shower. I don't think a lot of Americans would be willing to deal with that, but Japan has a public bath culture/history.
In Tokyo in particular, there's a huge amount of public transportation options, which allow people to live farther out and commute easily (with the commute costs being covered by work) -- that's not as simple in NYC. Imagine living in White Plains and commuting to the heart of Tokyo every day in a reliable, timely fashion.
Another thing is that a lot of cheaper housing in Tokyo is just old (post-war wood construction with a veneer of modernity) or in just plain bad areas (low-lying areas near rivers, below the water table, on reclaimed land, far from hospitals/fire departments, etc.)
On top of all that, housing here just isn't an investment like it is in America. Nobody's house or apartment appreciates with age.
It's not as simple as pointing at the average rent and zoning.