But I think there's some things we know we can assume are offensive. And there's other things that a person can say "stop it, that's offensive". And that creates an obligation on you under the general principles of good manners.
Now you might not have realised it's offensive. You might disagree - you think it's fun, and legitimately believe most people think it's fun too. But once you've been told to stop, it's pretty clear. It's perfectly okay to be corrected and to learn. And it's normal to behave different with different people.
So it hardly seems like there's a big issue here.