I've found the best way is just to accept the silly label if someone accuses you of "racism".
My and I guess many others point in this thread is that a lot of people are running around alienating a lot of people who would be ashamed to even catch themselves thinking a racist thought by insisting they are "racist" since they might have implicit biases.