"This argument seems like an arbitrary limitation of the human mind, which I don't think holds up. "

And why not?

Evolution did not reward us for thinking about spaced out concepts, but for coming up with new ways to get food, outsmart the prey, build tools.

That thinking in 4D is helpful for building tools is a new thing so we evolution did not optmized for it (yet).