Or scrolling represents a viewpoint, such as a frame, or a finger pointing at the document. Then it's the other way round again. You can't just declare what the metaphor is, it's arbitrary.
Which is not natural. The manipulation of a trackpad matches the manipulation of a document, natural.
There's no real world concept of a 'viewport' that makes sense with documents other than our own eyes; we don't move our eyes with our hands, unnatural.