It's all to do with how your brain processes what it's seeing, and the planning processes involved in getting to the next bit of information it needs. If that information is off-screen, then the mechanisms for stashing the current state and planning to move your hands in whatever way necessary to bring it onscreen will kick in, and that's a sort of disfluency.
Similarly with tokens too far from whatever you're currently focused on. There's likely to be a region (or possibly a number of tokens) around your current focal point within which your brain can accurately task your eyes to scan, and outside that, there's a seeking disfluency.
I think this is why you get weird edge cases like k and j, where they pride themselves on having All The Code in one 80x24 buffer, and it actually works for them despite breaking all the rules about code legibility.