It's reasonable and natural to have a mental model that the control moves the observer. (move a control up to aim your eyes up)
It's also reasonable and natural to have a mental model that the control moves the object. (move a control down to "grab" the object and move it down)
Both of these are natural and everyone does both in real life totally automatically without thinking.
Everyone looks up and down. Everyone grabs objects and moves them to bring different parts into view.
Probably the preference differences are based on a subconscious/unconscious difference in how you imagine yourself in relation to a document. Whether you imagine yourself as being larger than the document like a person vs a paper, you move the paper, or you imagine the document as larger than you like a fly flying over a paper or like you are virtually IN the document, you move yourself.