Having used vi over low-speed serial links means that I can handle latency just fine. Also hjkl were noticably faster because they sent fewer characters over the lines compared to arrow keys which used escape codes...
If I'm editing a file and type several repeating j or l commands (e.g., 'jjjjj' in command mode), there is noticeable latency between the last movement command and the last cursor movement.
I get no latency, however, when performing the same commands using the right or down arrow keys.
The 'h' or 'k' commands have no latency, either, just 'j' and 'l'.
I've been using vim for years and still find it frustrating. I haven't noticed the same problem with other vi clones (e.g., nvi or elvis).
(aside-rant: evil-mode is also not really a substitute for vim -- when you say you're a vim user, people often ask "why not just use the vim emulation in editor-X?" and the reason is because vim emulation has never, in my experience, been as good an experience as vim, things always work subtly differently)
I think there were two or three things that weren't right for me out-of-the-box, but 15 minutes and 30 lines of elisp later it was fixed. I really need to maintain a fork or something that is as much like vim as possible, but with options for enabling specific emacs features, because everyone has some things that emacs does that they like better than how vim does it.
My personal example: s/foo/bar/g on a line containing "foo Foo" becomes "bar Bar" by default in emacs, which I find to be amazing, but is definitely not how vim works.