You can reduce the delay by setting ttimeoutlen, the time in milliseconds between an escape key being pressed (or jk etc. that are mapped to Esc) and exiting Insert mode. I have it set to 100.