That's a very good point as well. You probably couldn't move enough matter to make a noticeable difference to the Universe as a whole, but it does point to a violation of some kind of equilibrium.

I think it's just easier to assume time travel is impossible.