One way to fight these biases is to remind ourselves that it’s a very small percentage of any demographic that commits violent crimes. It might be more useful, then, to say “99% of X people are never violent versus 99.2% of Y people.”
But to throw a hypothetical at you: is it right for an Asian person in San Francisco to be more scared when they see someone of "X" race than when they see someone of "Y" race?
Even if most of the people in the "X" group are nonviolent, if threats disproportionately come from the "X" group, it would not be unreasonable to be more wary of members of that group.