All around I see new apartments being built and I guess that's great, but I haven't seen anywhere near as many infrastructure projects in the same time.
In the bay area, we have good city planning for our primary infrastructure besides transit and housing. For instance, Sunnyvale owns its wastewater treatment plant and is prudently investing in it. Housing has no real planning, except for city councils (who benefit from a housing shortage) approving projects. Transit is a derived need that stems from our housing inbalance among other things.
If transit isn't working, ambulances can't get around and children can't get to school. If water isn't working, we can't operate hospitals. This is why we distinguish primary and secondary infrastructure as layers of abstraction.
Not to say that secondary infrastructure isn't important, but it doesn't have as much public sunk cost investment in things like the digging the networks of tunnels and such, so secondary infrastructure is easier to fix in a shorter time span.
city councils (who benefit from a housing shortage)
In regional planning however which tends to lump in every county and every city (as legally defined by California law) in those counties, San Francisco has essentially zero political power. It has one vote on matters, and the interests of the many cities and counties around don't always squarely align with what SF wants to do. To put it bluntly, even if San Francisco were the epitome of high governance and quality infrastructure planning, even if San Francisco had stack of workable solutions printed out on a perfect stack of paper with absolutely stunning typography and presented and overwhelmingly supported by quality leadership with a fully funded budget and electoral mandate, there would be nothing SF could do to solve any of what ails the region as a whole without the cooperation of the other cities and counties.
Homelessness in San Francisco and the larger cities? That's their problem. Lack of housing? That's their problem. Not enough regional public transit? Not a problem here! Everyone has a car!
So yeah, the free market fairy won't solve anything, but neither will regional planning boards. At least the State can put pressure on local governments and they can either react sensibly, or keep trying to push every problem back into the urban core and just bottle it up there. When the pressure builds up, maybe we can get the free market fairy and the socialist fairy to meetup at a bar and hash something out. Would be nice anyway but I wouldn't hold my breath.
* I'm being incredibly generous by allowing San Jose to call itself a city, but San Francisco is essentially the biggest fish in the region.