Imagine that you are walking by someone and accidentally elbow them in the ribs. They double over in pain, and exclaim loudly that you hurt them really bad and you should apologize. Now you could get defensive and say "Well I didn't mean to elbow you in the ribs, it wasn't on purpose so I did nothing wrong" and "Also getting elbowed in the ribs isn't actually that painful, there's no reason to make a scene about it."
But what you don't realize is this person was in a car accident last week, and below their shirt their ribs are broken. Your elbow just hit them on a sensitive bruise and jostled a broken rib. Your unintentional action was actually extremely hurtful. By protesting that you didn't do anything to hurt them you are just looking like a jerk. All it would have taken was an "I'm sorry, I'll be more careful" and it probably would have been fine.
This is similar to what racism and the effects of racism are: there are people who have suffered trauma from heinous acts. As a result they are already in a wounded state, and something "not racist" in your mind can still hurt because it pokes in the same place as the "actual racism". Just like you can't accurately judge whether someone is actually in physical pain and how much physical pain they are in, you can't always tell how much your action hurts someone emotionally.
In both cases the empathetic and moral thing to do is to believe the person in pain who says that your actions are hurtful.