Note that all symbols are their meaning are made up. To say "this gesture is a symbol of X" is to say "people made up that this gesture is a symbol of X."
I don't like it when folks say such things, but I understand it. Instead, what many folks seem to be saying is that based on one thing they can tell what the person is like inside. It's an amazingly-broad generalization and people seem to be enormously confident of themselves when they make it. I can look at your shoe or color of your shirt and conclude that you are a bad person inside.
I don't find that to be especially rational, but what is one to do?
The op is not wrong for supposing it's possible for someone to be targeted due to wearing "shoe or shirt color".
The topic was "amazingly-broad generalization".
Context matters.