I think this is different because I'm full on opening an instance of vim. (I didn't know about GNU Readline though and it looks awesome - I'll definitely try and work it into a project at some point).
Just so others know, you have to put bash in vi mode and press Esc before v (otherwise it's ctrl+x ctrl+e).
set -o vi