I wonder if it is an instinctive reaction:

a large new space has a lot of unknowns -> some of these unknowns could be dangerous -> a very large new space has a lot more unknowns -> very large spaces are to be feared.

This is based on nothing more than my imagination, but seems like a sensible evolutionary precaution.

I tend to think that it is instinctual, probably related to the vastness of the thing, and the fact that it is simply beyond our ability to grasp completely.