If you don't remember the past, slow down relative to what?
Another thought experiment: suppose reincarnation exists and, as soon as I'm born, I remember my past lives, would time go faster or slower?
I don't expect an answer, they are just thoughts that I have...