In my opinion the main culprit are modifier keys (e.g. by default the shell uses emacs bindings and you do a lot of ^A, ^E, ^R, etc.), but using jj mostly solves the issue as you just have to reach for Ctrl when you require SIGINT or copy/pasting in different environments that do not support vi keys. There are also people that remap it to Caps-Lock, but I just don't like to use my pinky that often.
Of course, everybody is different, but it makes sense to think about a setup that suits you and your health.