What about using a browser, terminal usage outside of vim, and insert-mode in vim? Do you use vimium in chrome and vim-mode in bash?
I use a “vimium” plug-in for Firefox and I quite like being able to go up/down with j/k. Though I still use the arrow keys for my terminal.
I've tried out vim-mode bash in the past, but much prefer the default emacs-mode on the command line, despite being a vim user.
Either way I'm suspecting a ton of exceptions which necessitate resorting often to the cursor keys, however (like another commenter has mentioned w.r.t various GUI apps).