Agreed! In my mind, to go back in time is to put all particles or whatever fundamental components back in the same place and state as they were previously. Thus, that would include the particles that make up you as well. How would you ever do that and if you did, how would you notice and is it really the past or just now with a replicated state?