> I’m not quite sure why it works this way.
It works this way because people are, rationally, more interested in their own physical safety than they are in the feelings of strangers.
Anecdote: I was in San Francisco a few years back. There were homeless folks everywhere. Some were nice, even chatty. We felt at ease even when walking through areas that were full of homeless folks. But a couple of days into our trip, a homeless dude accosted a coworker on the sidewalk. He shouted violent threats, shoved my coworker, and told him not to come back. When I asked my coworker what set the guy off, he said that he had absolutely no idea.
That experience changed how I interacted with homeless folks for the remainder of the trip. I made no eye contact. I avoided areas with groups of homeless folks. I did not respond when spoken to. Is that fair to the average homeless person, who is perfectly normal, just down on their luck? To be perfectly honest, I do not care. My physical safety comes before your feelings. Full stop.