Ironically -- likely on purpose -- that's the point. The series centres on an unreliable narrator, which is a part of the charm.
You forget moments because the narrator wasn't clear.
The narrator wasn't clear because the he didn't understand what he was seeing.
You have to re-read the series, perhaps several times, to figure out what's truly going on.
Long Sun is my current favorite of his, but you should be warned that it’s a very different kind of writing and pacing. The entire first book takes place over the course of only two or three days, and you’re given even less context about the world than in New Sun. The reward for puzzling it out is even bigger IMO, but I’m in the minority here. (And I’ve by no means caught everything.)
He's not just writing, he's constructing beautifully crafted puzzle-boxes that take a lot of time and thought to unpack.
Couldn't believe I spent so much time reading SF without having heard of this.
The societal descriptions were truly amazing though. Coming out of the despair and decline with some false hope that humanity’s physics was good enough to face off and then absolutely getting their ass handed to them in a conflict with what was essentially something less than a messenger probe. I’m absolutely convinced this is what we will do as a race in such a situation.