You were doing it right the first time. Vim's "insert mode special keys" are basically incredibly basic terminal controls that are the same as Readline anyway (like backspace, or ^U to delete to the beginning of the line), or incredibly obscure things like inserting the contents of a specific register. Vim doesn't expect you to move the cursor much in insert mode, it's just for typing.
As it turns out, the extra Readline bindings this emulates are basically emulating Emacs' standard bindings, so if you want those bindings in an editor, you know where to look.
Personally, I just stick these settings in ~/.inputrc:
set editing-mode vi
set keymap vi
...and then every application that uses Readline (including bash, the Python REPL, the PostgreSQL REPL, and a bunch of other things) defaults to vi keybindings. It's dreamy.