You'll also want `set show-mode-in-prompt on`.

Note that this feature is broken when you have a PS1 with a \n in it, in an older version of readline/bash (I am not sure which), so if you're having this problem, just upgrade your bash version.