I would argue that factual past is unprovable without some sort of visual evidence, and even that can be manipulated...especially with AI on the rise. The problem with your point is that EVERYTHING is parsed through your perception, and you can just as easily make the point with the same logic that you can't prove that what you are currently experiencing is present reality.

What actually happened is unprovable without some layer of trust once the event leaves the affected. For example, I ate a grapefruit for breakfast. That is a fact. However, I have tossed the peel away and I am communicating that I ate it with a stranger over the internet. For all you know, I could have eaten cookie crisp. If you and enough people get together and collectively believe that I ate cookie crisp, the public belief will be that I ate cookie crisp. However, that does not change the fact that I ate a grapefruit.