If I'm editing a file and type several repeating j or l commands (e.g., 'jjjjj' in command mode), there is noticeable latency between the last movement command and the last cursor movement.
I get no latency, however, when performing the same commands using the right or down arrow keys.
The 'h' or 'k' commands have no latency, either, just 'j' and 'l'.
I've been using vim for years and still find it frustrating. I haven't noticed the same problem with other vi clones (e.g., nvi or elvis).