The root cause seems to be the same as with most of humanity's problems: overpopulation.
There shouldn't be any population growth to keep up with. The problem is too many people on too little space. Not sure why people seem to want to live in cities that resemble ant-hills if it's shown that it makes them miserable and depressed.
And still be absolutely miserable. Dense housing it part of the problem, humans need space, and lots of it.
Also, it is a local solution, if SF had 1000 times fewer people current problems wouldn’t exist either.