The fundamental reason for this is simple. Humans are prone to cognitive dissonance. Meaning, we do absurd things to avoid painful thoughts. And anything that questions our sense of identity, is a painful thought.

So if my self-image is, "I've advanced our understanding of the fundamental nature of reality," then the idea that my contributions weren't useful becomes painful. So we avoid thinking it, challenge people who question our past contributions, and so on.

The natural result of this cognitive dissonance is a feeling of undue certainty in our speculations. After all certainty is merely a belief that one idea is easy to believe and its opposites are hard to believe. We imagine that our certitudes are based on fact. But they more easily arise from cognitive biases.

And this is how a group of intelligent and usually rational people descend into theology whose internal contradictions can't be acknowledged.