The problem is that if you don't personally know the views of the people involved, racist jokes are often indistinguishable from hate.

Outside of a context where the humor is understood by all like a stand-up comedy act, you aren't entitled to how other people perceive your behavior.

If you walk like a racist and talk like a racist, people are just gonna assume you're a racist.

If that's a problem for you, there's a really fucking simple solution.