So, let's apply the process. Suppose Person A says something, and Person B thinks another colleague, Person C, might find offensive.
That alone may be enough to offend Person B. They feel Person A may be creating an uncomfortable environment, intentional or not. Person C may not even be actually offended, but other people of Person C's {race,sex,etc} have been offended by similar statements. How people apply judgment is an individual process.
Should we care about how any of these people feel? Is Person {A,B,C} right or wrong to feel how they do? We can't control how other people feel about things, but we can try our best as individuals to promote welcoming environments, but recognize we are human and won't always get it right.