When you replay a series of inputs, you want the accompanying physics to come out the same. Solvers that use randomness in your engine will either need to have their RNG seeded with the same value or have the rng removed.
Another thing causing problems might be how your engine handles time or when it iterates over objects in a specific order relative to how they were added (might not have preserved that order in your snapshot).
There's many things that can go wrong.