If you go back in time but don't do anything at all in order to not change any event from occurring the way its meant to then I guess it could be like that. But if we assume there's only 1 timeline and we change the past then the current future changes, you wouldn't know that you changed the future because the timeline you came from is the timeline you altered. But if you altered the timeline preventing you from needing to travel back then you would never travel back to prevent the timeline from occurring.
The timeline would shift in such an odd way too, because the past and future co-exist with each other, if you changed the past to fix the future the moment you travel would alter the timeline instantly and possibly prevent the need to ever travel in which case everyone would be shifted all of time from the moment the change in the past occurred...
It becomes very confusing, think I need a migraine pill and some sleep now...