"The honest truth is that you shouldn't. " (believe your own memory)

That does not seem like a good way to live.

Imagine "not trusting your memory" tomorrow morning, and refusing to recognize your bedroom, your spouse, your kids, family, your house, car, coworkers etc.

Seems like one would go insane, quickly. So yeah, I guess you do need to trust your memory - but judiciously.