Like this saying "There is nothing in the form of the laws of nature at the fundamental microscopic level that distinguishes a direction of time." There is nothing in the simulation which prefers to step time by 1 or -1 , but the unavoidable rounding error means it can't step exactly backward. I guess its intimately attached to the fact it cant step exactly forward.
This is why the shattered virtual object wont rearrange perfectly going backwards - it didnt smash perfectly when it was going forward.
I had an idea for making previous states recoverable by applying extra rounding between every timestep, but now I suspect this will not entirely work either, as some least significant imprecision occuring during the timestep will still occasionally trip the more significantly rounded 'interstep' state values, irrecoverably.