It stems from a few things, and novelty is only one factor. (Also, it is not only that the world begins (to us) as full of novelty, but also most people gradually transition from exploration to exploitation.)
In addition to remembering previous events, there is also how fast we process information (e.g. reaction times). It seems that as we are getting longer, we get fewer clock ticks per second. (On an interesting take on that, read a short story "Exhalation" by Ted Chiang.)