As someone who uses Vim daily, I quite like this idea. I would customise it to mimic the original Vim bindings a bit more closely where possible though, to e.g. keep Ctrl-D as page down.

I'm not sure if this would be possible, but it would be fun to have customisable per-application bindings (based on the active window, probably) in addition to the set of global bindings. I can also think of minor cases where motions would be useful, like 3x alt-tab (i.e. 3gt) to switch to a specific application.

I wonder at what point the depth of the mode-ness will become too much, if you were running this, for example, with a vi-mode bash readline.