I told my mom about it decades ago, and she said there was a clown that jumped over the fence. Then I spent the rest of the party running around the house holding the balloons.
There's also the phenomenon of having a memory of a memory. At age 10, I had a very solid recollection of my life at ages 5 to 6 (not so much of age 4). Now all I remember is that I used to remember a lot more than I do know.
I have exactly the same thing. It can get very detailed, too, but it really feels like remembering reminiscing about those memories at like 7-10.
I kinda suspect this is true for a lot of people, but since memories aren't time-stamped, they don't realize how early they are.