I feel like such a tool for saying this, because I thought people who said this were tools before I 'got it'. Taking your fingers off the keyboard and to the mouse and trackpad is slow. I now even get how moving your fingers to the arrow keys is slow. I never thought I would believe it, but somehow here I am, spouting the same nonsense.
It's true. I think there is just some mind-body-editor connection that develops with vi(m) usage (and emacs, I don't know). When you really start thinking in vim, your mind starts describing what you need to do in vim commands, that magically flow out of your fingers, and vim responds.
In other editors, for any task I need to do, instead of telling my editor to do something, I have to show my editor how to do it by selecting text and menu items with the mouse, it just feels clumsy.
Once the keystrokes are muscle memory, it's impossible to beat with point and click in terms of speed.
That's just an opinion from a vim novice that can never go back.