This is not at all similar.

The correct analogy would be if the person I accidentally elbowed claimed I was physically assaulting them, which I of course would deny.

Racism is a very extreme call-out and it is no wonder that people defend against it. It is no matter of believing that I hurt the person, I do. But I am defending myself from an attack I believe is unproportional,and actually leaves no choice but to defend myself. Call it insensitive instead of racist and people are far more likely to listen.