This makes sense evolutionarily. I imagine that for the primitive man, a concept like time did not make much sense -- position and motion were the important things to track. We were aware of change, but perhaps we did not conceptualize of change occurring within a stratum like time. Were there any reasons we would track the number of sunsets before the advent of agriculture?
I would really like to see some research on why time behaves so strangely on psychedelics. Our everyday consciousness is mostly stable and steady so it's quite natural to think of it as dependent on time, as if there is a constant metronome quietly ticking away in the background. Most people assume that 2018 will feel mostly the same and take about as long as 2017. Only when things start slowing down and we peer behind the curtains of our egos do we get the sense that things are much weirder than we'd like to think.
Personally, I think the illusion of time must involve a many different brain processes. Memory is certainly one of the keys, but I wonder if the overall state of connectivity or speed or processing of the brain doesn't have an effect. The ego also seems implicated somehow, as we experience time differently in states of flow and moments of awe and wonder.