All of what you describe is probably a big part of why we don’t remember. Not only has the though encoding/decoding process not been developed sufficiently to remember things in the way we do when older but even if we did those memories would probably be horrific for the most part.