Yes this is my favorite explanation: that consciousness is incredibly brief and delicate. That only one very precise organization of the brain could give rise to the epiphenomenon that is me. So precise in fact that "I" only exist for a very brief moment.

What I perceive as a continuous consciousness is really just a memory of other "I"s who were caused by this brain in the past, left a memory of introspection and then dissipated to be replaced by an new one.

This view has many advantages: it solves the problem of time you elude to, it answers the problem of other minds and the problem of my childhood self. The only problem I can not see solved is the hypothetical molecular level duplicate I mentioned above.