You don’t understand fondly reminiscing about when the world seemed more magical? It has nothing to do with the truth of it, it has everything to do with how it felt.
So too with knowledge. The path to never believing anything false would be to simply minimize the amount you learn, and certainly to avoid learning anything developed within the past century or so. It's not a great perspective for self betterment. Learning from one's mistakes is the path to improvement in most of every domain. Mistakes and falsehoods are nothing to regret. If anything, they're something to rejoice over. It's only the fool that believes all he thinks must be true.