Too much shit on the street, for starters, not to mention it's definitely more dangerous to walk, especially at night. Lived there for 15+ years. I had to move, the rent was not justifiable, and it was clear the problems were getting worse. shrugs
And the areas I've lived in definitely have massively more "SF culture" than downtown.