If you're in the Tenderloin, Civic Center area, parts of the Mission or SoMA, then sure, you know you're in a dodgy area. But I'm pretty sure that if you're in Forest Hills, West Portal, Lake Merced, or the Presidio, you'll feel pretty safe, and crime maps seem to bear that out. I'll take my preschoolers alone to visit friends in the city on public transportation - I drive to Forest Hills, park in the neighborhood, muni to the Castro. Never had a problem.
I think that what gets tourists to SF is that areas you'd expect would be safe are actually bad. Normally you'd expect that major shopping districts like Union Square, or the city government like Civic Center, would be safe. In SF they're right next to the Tenderloin, which is a cesspit of filth and addicts. And gentrified vs. dodgy areas in SF are very close together, like walking two blocks toward Valencia and Guerrero in the Mission is dramatically different from walking two blocks in the opposite direction to Van Ness and Folsom, or being on the north side of Potrero is dramatically different from wandering the south side.