It seems that the 'root cause analysis' is many times simply superficial. Too many homeless people? Give them homes. Too many addicts shoplifting to fund their habit? Decriminalize shoplifting. Cost of rental housing too high? Impose rent control.
This may be a consequence of the amount of time a politician is in office vs. the amount of time it really takes to solve these intractable problems. If a politician's term is 4 years, and it takes 12 years to solve a problem, actually identifying the root cause and devising a solution would be the end of their career in politics.
I'm pretty sure that if San Franciscans could elucidate what kind of a city they want San Francisco to be, no one would say they want it to be a haven for drug addicts and have the world's highest rents. And there are enough neighborhoods that just about all environs they'd list could be implemented somewhere within the city limits.
Given that San Francisco is not the kind of city that San Franciscans want it to be, I can only chalk it up to ineffective leadership by the persons setting and enforcing the rules. That's not solely the politicians. I'd guess it starts with leadership trying to satisfy everyone's issues without triage for practicality or an investigation of possible unintended consequences of their solutions.