There is literally no way for you to know that, because the facilities you would have needed to know it only grew in later. Your memory is not at all definitive here, because we make up memories to make things make sense, at this level.
I'll take a stab and say that at 3-4 years old I had a pretty decent idea that other people had a mind of their own. It wasn't so easily learned but oddly enough I picked it up around 3 that I knew my mind was wholly separated from my sister, mom, dad, and grand parents. I can't explain how I got the concept down I just did. A better way to explain it is that it's also when my memories become far more concrete was around 3 years old (about the time I got into Transformers and He-man. An odd way to place it but it was definitely when things got more like what they are now, before then it was a mushy blur of noise and color tbh.).
That's what you say, but the argument is circular. Not saying you're wrong---but you're not obviously right.