We attach our early memories to our concept of the world as it was then. Afterwards, our concept of the world changes. As a result, those old memories sort of become dangling pointers. They refer to a world model that no longer exists in our minds.
That's just aggressive garbage collection happening when you wake up, arguably so you don't risk confusing dreams and reality :-)
But you can somewhat partially work around it by taking notes right after waking up, before doing anything else.
I would hazard a guess that the original pointers aren't revised to point nowhere, but are, if possible, moved to point into the revised reference model. (Presumably we do this now when we learn a new domain like music, and as the model for harmony becomes more concrete, our original pointers to elements within it are adjusted rather than lost.)
In other cases I believe it’s quite possible that our understanding of the world, say at age two, is so radically different from our adult understanding, that the old memories don’t even make any sense in the new context. They can’t be remapped in any meaningful manner.