Do you mean to say that our experience of time depends on what we see happening around us? If I place you in a room with nothing but a video tape playing backwards, would you experience time going backwards? That doesn't sound plausible.
What I mean is that since our brains store memories and generally process information utilizing entropic systems, we perceive time as always moving forward. There is no fundamental physical direction to time... But since our experience slides along an axis of increasing entropy, we perceive the whole universe sliding on that axis as well.
It just doesn’t matter on a universal level.
Kurzgesagt – In a Nutshell has a very good video on it:
Did The Future Already Happen? - The Paradox of Time
"To be honest, no one knows. What we’ve learned are two possibilities to describe time, but they're not the only ones. Some scientists think that the idea of “now” only makes sense near you, but not in the universe as a whole. Others think that time itself doesn’t even exist – that the whole concept is an illusion of our human mind. And others think that time does exist, but that it's not a fundamental feature of the universe – rather, time may be something that emerges from a deeper level of reality, just like heat emerges from the motion of individual molecules or life emerges from the interactions of lifeless proteins."
They touch on block universe: https://en.wikipedia.org/wiki/Eternalism_(philosophy_of_time...
…and growing block universe: https://en.wikipedia.org/wiki/Eternalism_(philosophy_of_time...
"The growing block universe, or the growing block view, is a theory of time arguing that the past and present both exist, and the future as yet does not. The present is an objective property, to be compared with a moving spotlight."
Excellent video worth your time.