An acceptable approach would seem to be to have the "center" becoming more dense over time while the outer rings stay less dense.
The problem is that current geography has resulted in natural centers being the ones that refuse to build and that is problematic. Palo Alto and the rest are happy to have Facebook and etc's business but won't make room for Facebook employees.
And the apparently illogical behavior comes out of proposition 13's logic. Cities can tax businesses but property values are immune.
Which is to say that prop 13 needs to be repealed or annulled before any other sort of sanity is going to happen (and maybe that's not happened but, hey, there you go).