That's not really the same thing though is it? Each of vim's modes are extremely complex with their own usage of the vim grammar. The "modal" behavior you describe is very simple.
The main difference is Vim's editing modes are imperative. You have to glue together small painful commands to do what you want. Modern editors are declarative - you say "I want to move this file" or "I want to rename this variable" or "I'll drag this split here" and they do the rest.
Those are imperative actions rephrased to make an artificial distinction (and I rename variables in Vim with gR). How high-level the available actions are and being a modal or non-modal editor are orthogonal.
Painful?
Physically or mentally?
Physically I find Vim to be pain-free. I've studied and I practice good typing form, which is important—I would look there before blaming any physical pain on Vim's commands.
Mentally I find Vim's interface to be beautiful. Learning it gives your mind an amazingly coherent structure for thinking about the process of text editing.