Part of the different perceptions really depends on what part of SF you spend your time in and what parts you avoid.
Yeah the city is only a 7x7 mile square. But block by block it can be dramatically different.
Check out the map of violent crime halfway down this article: https://sfstandard.com/criminal-justice/bob-lee-killing-viol...
I lived on Nob Hill for 7 years, it was fairly quiet, always felt safe. We had one unhoused guy who was harmless, you'd see him around.
Go down the hill, and cross Geary and you're in the Tenderloin. Violent crime on that map goes from 64 to 361. People are passed out on the sidewalk with needles hanging out of their arms. There are tents, trash, squalor everywhere.
Go over the Pacific Heights and it's $10-20M houses everywhere.
The Richmond and the Sunset are like any suburbs, just a tiny bit more dense (houses are touching but they often have back yards). Low crime. Sleepy.
After Nob Hill I lived in SOMA for 5 years, just half a block from 6th street. I once counted 12 tents on my block. A guy was tapping the street light for electricity. Needles in arms, people screaming, the sidewalk is a bathroom. Go east of 5th street (or maybe east of 4th) and it's all modern buildings. In the time since SalesForce tower went up, that area completely transformed. It's hyper modern full of high end office and residential buildings.
(This is the area where Bob Lee was killed
When I lived in Nob Hill, I felt like I was in this great dense walkable urban neighborhood. Then I made the mistake of moving to SOMA, it was nice that I was close to so many tech jobs, but being exposed to what went on there made me grow to hate San Francisco.
A lot of people just actively avoid those really bad areas. So they never see it. Unfortunately I either lived in the middle of it, or regularly walked through it.