I think you misunderstood. The point is that it's not in his head - he knows his Indian colleagues are trying to figure out his caste. Non-Indians don't have a caste so he knows it's safe in that situation.
After you endure enough discrimination and persecution, your trust in others are eroded. You start to irrationally question the motives of others. Eventually you lash out preemptively because you’re afraid.
This is why discrimination and oppression is bad. It tears at the fabric of society.