There are a great many things you can't debate about people. For example, you would be severely censured if you debated whether your colleague's chronic illness really was as bad as they claim it is.

If the position is "I should be able to question and debate anything about my colleagues" then the position is obviously absurd and I would ask a person holding such a view to reevaluate what they think society is. I don't think you would hold such a position.

So, why are you troubled that you can't debate this particular thing, when I'm sure there are things you shouldn't debate about your colleagues?