Time travel is entirely outside of our theory of physics.
If you want to insist on rules, you have to say that we can't do it, not argue about how it would work.
(I mean, it is entirely possible that the universe is a-causal in the presence of certain types of events that we do not yet understand)