I'd wondered why hitting 'O' after hitting escape led to such a nasty delay when running vim in a terminal. Now I know.

For those that still want arrow keys in insert mode (sacrilege! ... but I do), this is probably the .vimrc option you want:

    set timeout timeoutlen=1000 ttimeoutlen=100