I understand the mental stress of having people after you all the time, but that doesn't mean that all the people are after you all the time. In this case they may have extended a genuine invitation. And given the historic context, it may have led to him feel like it was a casteist action, that doesn't make his colleagues casteist.