I get what you're saying. But the physicality of a distance seems more meaningful than our movement through that distance.
I've tried articulating this idea before but its very cumbersome: You want to go make a drink in the kitchen, the difference between the future you making that drink and right now is only your movement to get to that point. You making the drink and you not making the drink are exactly the same "points in time". But for perception, you making the drink is an idea you see as a future event as opposed to a concurrent event that involved some movement through a distance.