It’s hard even when you don’t take into consideration that you don’t know what you’ve misremembered. Try writing a memoir. You’ll realize you never actually remember what anyone says, but caveating your dialog with “and then I think they said something like” would make horrible reading.