So shouldn't we expect there to be millions/billions of people interfering with the past. A significant percentage of which would be deliberately aiming to manipulate, destroy, communicate, affect and do all the things we expect people wouldn't.
Clearly it means either two things. 1: time travel is not possible. 2: going back in time forks the universe/space-time (About Time is a recent movie that proposes this).