I can flip that around and say that a distance is just the movement of something over a measured time. By your original argument, distance doesn't exist.
I've tried articulating this idea before but its very cumbersome: You want to go make a drink in the kitchen, the difference between the future you making that drink and right now is only your movement to get to that point. You making the drink and you not making the drink are exactly the same "points in time". But for perception, you making the drink is an idea you see as a future event as opposed to a concurrent event that involved some movement through a distance.