I'm from New York and frequently travel to China, both of which obviously have "old" spots. For some reason, "old" there for me evokes the feeling of appreciation for history, while the San Francisco "old" makes me sad.
Market Street around civic center is appalling, and it spills over into Powell street and the Embarcadero near market enough that those spots are clearly tarnished, especially in the eyes of visitors. But pretty much every corner of the city suffers from blight - the new Academy of Sciences in Golden Gate Park or the De Young may sparkle (a little too much for my taste), but the appeal of the park is diminished by the presence of a large homeless population (including stray pit bulls). Maybe these things are more noticeably to newcomers now?
While it may seem a little facile to blame too much on one problem, I don't think you can understate how severe an impact the "hardcore homeless" population has had on SF. Not only does it create blight, it's also a big drain on our finances. SF is in a tough spot here. It's a liberal city that has friendlier homeless policies, but unfortunately, that makes us a magnet.
As for New York... lived there for only 6 months, but Mahnattan is truly a vibrant place. I was at Columbia, and I'd say I was hit up for spare change about as often as I am in SF, though. This was in the mid 90s, maybe it's cleaned up better than SF since then, haven't really been back.