TBH I think I almost never look at the line numbers in vi, but the alternative isn't always holding/mashing j.
I occasionally use a count (5j to move 5 lines down, especially good for moving ~4 lines at a time), sometimes use C-u/d, C-f/b, or H/L to scroll half a page up/down, a full page up/down, or move the cursor to the Highest/Lowest line on the screen (these are good for large movement. And you can use zz to recenter the cursor on the screen after H/L). But likely the most common vertical movement I use is curly brackets/braces to move a paragraph up/down.
In CUA editors to me the nicest shortcuts are Control[+Shift]+Backspace and Control[+Shift]+Arrow because they involve the editor understanding a basic construct of human language (words). Vi takes that to the next level.