Exactly. No one intentionally drawing attention to themselves could possibly be up to something nefarious, they'd draw too much attention to themselves to get away with anything.
I think GP was playing on the assumption that those employed in these traditionally low-paid jobs such as security and maintenance would predominantly be black.
I once was shadowing a deliver driver in a suburban neighborhood. We were both in high viz and I stayed in the car for this delivery. The guy rang the bell and no one answered so he followed standard process and looked for a secure location to reduce the chance of theft when he left the package unattended. He was looking for just a few seconds when a neighbor came up to hassle him, ask him what he was doing and accused him of wanting to rob the place. While wearing a hi-viz vest. Racists will see whatever they want to see. He could have been wearing a flashing hard hat too and I'm sure the neighbor still would have harassed him.