When I first arrived in Austin a few years ago, so many yuppies told me to NOT go east of I35. They were paying outrageous sums of money (actually quite tame by San Francisco standards) to live far from the side of town they perceived as dangerous.
However, once I became acclimated to the city, I realized that they were blowing a lot of smoke due to irrational fears. The east side was certainly more diverse. It wasn't as white-washed as the west side of Austin. It has more graffiti and was generally poorer. But, it was not dangerous.
Again I'm making some pretty broad brush strokes. I realize that other places have dangerous neighborhoods and many cities have a legitimate need for a safety premium. My main point is that for many people, the necessity of the safety premium is perceived unjustly.