We live in a democracy. The underlying problem of government is voters.
Most home owners (especially in The Bay) refuse to allow anyone to build anything because they see that as a threat to their home value.
Building with x% percentage affordable units in it? Not enough affordable units and the units are not affordable enough. This will not offset the gentrification caused by the market rate units.
Building with high percentage of deeply affordable units in it? You can't just solve societal problems by throwing housing at it. The residents who will live in these deeply affordable units need social services to get their lives back on track.
Building with 100% deeply affordable units with on site social services? You can't put a homeless shelter in our neighborhood! The honest working class residents of this neighborhood have suffered enough from gentrification without having the homeless problem dumped on them too!
Pretty sure dictatorships have the same problem with improper zoning.
I can't find it now, but there was a study measuring the straightness of the road between capitals and their airports. The straighter the road, the more authoritarian the regime that built it.
It sounds like Bay area residents are working against their own interests (home values) by reducing housing supply, thereby making their city so unlivable that home values will collapse anyway. Getting people to see the problem and vote accordingly may be difficult, but it doesn't seem completely impossible.