However, my state of non-existence ended at my birth. Just as my state of existence will end at my death.
In this regard, the future is not exactly like the past. In the past, there was never a time in which I came into existence. In the future, there was a time where I came into existence. Neither of those statements transfer between past and future.
This is "memory", or "trace". Their existence take away from your non-existence, making it incomplete (imperfect), which indeed makes one's future different from one's past.
I'm not sure of that, yet. We haven't managed to pin down what "I" is or even what "existence" is. I'm thinking in the "we virtualized a copy of your brain" sense - just what is consciousness?
Indeed, the very existence (and non-existence) could be relative - just like time and space. For example, in one frame of reference there may exist a magnetic field while in the other there may be none.
From a purely entropic and information theoretic standpoint, these are very different states and certainly not a safe assumption.