To offer a counterexample, I have aphantasia, but have very detailed memories of my childhood. I have no problem recalling details like the full names of all my elementary school teachers, when I met most of my friends in school, and details about field trips we took.

That being said, I only have visual aphantasia, so it may be that remembering these peoples' voices or what places smelled and sounded like may be helping those memories.

I dated someone without aphantasia who was amazed at my childhood recall compared to her own. It turned out she later discovered she had SDAM.