We have partial impressions of the past. We have seen, felt, thought something at various points in time and then we construct a sequential story of what happened, often employing abstract cause-effect reasoning. Two different observers might come up with different stories. To reach a consensus, they have to exchange their points of view.
Once all the observers forget their story or they disappear, the past "fact" disappears from collective memory, unless that story was passed to newer generations by speech, writing, etc.