Here's a thought experiment, similar to the one-electron universe thought experiment: there exists one particle, the "consciousness particle," which randomly jumps around throughout 4D space. Whenever it appears at a 4D position, consciousness is experienced as computed by integrated information theory (or a similar model) [1].
When you experience a moment of consciousness at a specific 4D position, you experience all the memories of the "past" that are in your brain. Thus, the consciousness particle interpretation of consciousness is compatible with your direct perception of living a 3D life along a time dimension.
[1] https://en.wikipedia.org/wiki/Integrated_information_theory