I suppose it depends on how one's mind works. If the mind moves slowly and holds onto ideas and abstractions easily, then the slower motor movements of moving the hand back and forth from arrow keys to home keys, and to mouse and back, are not really a problem. But if you're like me, and your mind is constantly racing, you need to be fast to get your ideas out before they're gone/replaced.
It's not just the time cost of moving the hand from home row down to find the correct cursor key, do the presses, and then move back to typing position - it's also the likely visual/mental cost of looking away from your screen to aid in positioning the hand, then looking back at the screen and finding what you were looking at before. All those movements and recalibrations take time and involve mental context switches.
If you're spending a lot of time using a certain set of tools, there is no reason to avoid learning those tools well. Refusing to do so is choosing to be ignorant and subperformant.
Now, if you'll only use the tool once in a while, perhaps it's not worth the effort. But chances are, there's some other tool you use often that you could master for your long term benefit.
If you program, Vim (behavior) is very difficult to beat for getting whats in your head out and into your computer VERY quickly. As a bonus, using a full featured IDE like one of the JetBrains products, but with the Vim plugin, is win win. You get the code editing speed of vim, plus the project navigation and other benefits of the IDE. Totally worth the time investment.