That's an interesting way of thinking of it. I suppose this reveals an inherent problem with using statistics to represent crime — their significance is easily distorted, even if the numbers themselves are accurate.
But to throw a hypothetical at you: is it right for an Asian person in San Francisco to be more scared when they see someone of "X" race than when they see someone of "Y" race?
Even if most of the people in the "X" group are nonviolent, if threats disproportionately come from the "X" group, it would not be unreasonable to be more wary of members of that group.