Is this simply because the coefficient of friction between two surfaces is typically smaller when the surfaces are in motion than when they are at rest? If you wanted to reverse the insertion of a peg into a hole, you would have to halt it's motion for an instant, and at that point in time I would imagine you have to overcome static friction to make it move in the opposite direction. Friction forces can be dependent on the normal force between the two surfaces, so it's possible to jam a peg into a hole so that you can't get it out.