I didn't like that option, because I found I tend to hold Ctrl to do some action (maybe copy text from some panel), and if I change my mind and release it, the ESC might close the window I'm looking at.
It's happened more than once with Chrome's Inspector or in the shell. It would be even worse with Emacs in terminal mode, where ESC is a prefix key.