You can't know that.

I had memories as a toddler that I confirmed after meeting my mother for the first time in decades a couple of years ago. Specific things about where I lived, events that had happened and how someone else in our family had died. I had another conversation with an aunt that confirmed other things, including a word from another language that I knew from when I was little but didn't realize wasn't a made up word.

Memory is fallible and we have an imperfect understanding of it, but I know for a fact from personal experience that people can remember events from very early in life if the emotional impact is deep enough.