I lived in two of those areas for several years, and still saw more scary things happening on the street than I have in significantly longer time periods in other major metro areas. (The biggest difference fwiw was whether I lived on a very steep hill or not; the hill reduced the amount of street-side madness that I saw to near 0).

The clean and safe neighborhoods of SF are livable (I loved living there), but at least personally I have found them to be notably sketchier than any other city I've ever lived in and I agree with you that the contrast is not particularly close. >95% of the shoplifting I've personally witnessed in my life happened in SF despite it being a relatively small part of my total life experience.