Bret mentions Larry Tesla (starting at about 38:10) who made it his personal mission to eliminate modes from software.
This is the problem I've always had with Vim and I suspect I'm not alone in this. I find the concepts of modes jarring, even antiquated. Everyone who has used vi(m) has copied and pasted text in while in command mode and done who knows what.
Emacs is better in this regard but I find Emacs's need to consecutive key presses (or a key press followed by a command) to be longwinded.
The advantage of either is they're easy to use over ssh+screen (or tmux) for resuming sessions.
That all being said, give me a functional IDE any day. IDEs understand the language syntax. Vim can do a reasonable job of this. Emacs (with elisp) seems to do a better job (or so it appears; I'm no expert) but IDEs (my personal favourite being IntelliJ) just make everything easier. Things as simple as left-clicking on a method and going to its definition.
For statically typed languages (eg Java/C#), IDEs (quite rightly) rule supreme. Static code analysis, auto-completion, etc are just so much better than text-based editors.
Dynamic languages are more of a mixed bag and its certainly the norm for, say, Python and Ruby programmers to use one of these.
Still, give me an IDE any day.
One objection seems to be that people don't like using the mouse. I tend to think the speed differences over not using the mouse are largely illusory.
Anyway, I can use vim but I've never felt comfortable in it and I don't think I ever will and modes are the primary reason.
EDIT: I realize there are plugins and workarounds for many of these things but that's kinda the point: I don't want to spend hours/days/weeks/years messing with my config to get it "just right".
Also, instead of archaic commands to, say, find matching opening/closing braces (which tend to involve an imperfect understanding of the language), IntelliJ just highlights matching braces/parentheses, where a variable is used and so on.