It works this way because people are, rationally, more interested in their own physical safety than they are in the feelings of strangers.
Anecdote: I was in San Francisco a few years back. There were homeless folks everywhere. Some were nice, even chatty. We felt at ease even when walking through areas that were full of homeless folks. But a couple of days into our trip, a homeless dude accosted a coworker on the sidewalk. He shouted violent threats, shoved my coworker, and told him not to come back. When I asked my coworker what set the guy off, he said that he had absolutely no idea.
That experience changed how I interacted with homeless folks for the remainder of the trip. I made no eye contact. I avoided areas with groups of homeless folks. I did not respond when spoken to. Is that fair to the average homeless person, who is perfectly normal, just down on their luck? To be perfectly honest, I do not care. My physical safety comes before your feelings. Full stop.