Given that most people would care if even a stranger were assaulted and pushed to the ground, perhaps the clerk reasonably assumed you didn't care because it was a Catholic figure who was the target. In that case, he would be right in assuming YOU were intolerant - of Catholics.

Well, at least people around you must be tolerant of fat people.