I don’t have the details but I’m pretty sure passage of time is due to a tendency for entropy to increase, and that if I go back in time a second by, say, walking backwards, it is no more a violation of the cons of energy as if I walk forward.
So isn’t the simple layman’s answer that the “system” also spans the 4th dimension and you are just displacing mass around in a 4D space instead of the typical 3D space?