> I notice no 1 second delay when switching modes with the Esc key, so perhaps 
  > this is something you can configure in your terminal.

This is configurable in Vim. ttimeoutlen, timeoutlen, and noesckeys are pertinent. tmux et al may also need tweaking.