People seriously overvalue "being right". Discovering you're wrong is far more valuable and is worthy of celebration. It means you learned something, and are just a little less ignorant as a result.
Sure, but if in the course of being convinced of the wrong thing they have caused harm it should cause guilt and embarrassment.
I think that regardless of whether or not someone's belief is correct, causing harm to unconsenting others should always cause some amount of guilt or embarrassment.
But if I hold a belief that is leading me to cause harm to people, and then I discover that I'm wrong, is that not a reason for even greater celebration? It means that I have become a less harmful person.
That said, sometimes one needs to declare a position without be absolutely sure and it is wise in that scenario to declare the fact that you are unsure.