FWIW, I think the aversion to IDEs is bogus -- unless it comes from people who don't install plugins on their Vim/Emacs. If they do, then their problem is not IDEs per se, but how popular IDEs are implemented.
The gist of an IDE is "having all the functionality in one place": edit, search, refactor, browse code, scm, REPL, build, etc. People loading Vim/Emacs with plugin try to build the same thing essentially.
The problem with IDEs is not their "IDE-ness" but: 1) they lack a decent, fast, editor component. 2) they are slow to load and memory hogs.
Nothing of the two is inevitable. Just an issue with current implementations. For (1) we can image an ide with a full Vim component, for example, as it's editor part. For (2), the problem is that both 2 popular IDEs (Eclipse, Idea) are built with either Java (which is crappy for GUIs), or have a messed up GUI like Visual Studio.
Honestly IDEs in general are the most complex pieces of software that I frequently encounter. Most of that complexity comes in the form of features that I either never intend on using or don't intend on using at the time, which makes that complexity really frustrating. Complexity where it is necessary is fine of course, but I feel like IDEs like Eclipse frequently cross that line.
The whole point of integrated software is to be able to do everything in one place: debug interactive in editor, hover to inspect values, edit the code and have the ide recompile and reload the changed program.
The IDE that does that likely does other things, but having unused features should neither add complexity nor incur performance costs!
The complexity of setting up a dev env like the above in emacs or vim is absolutely staggering, but the complexity lies in the configuration and setup, not in the finished env, that's the difference. Personally my tolerance for "configuring" my tools (meaning gluing, telling my editor and profiler where my debugger is) is nearly zero.
As you say, Vim's complexity is different. It has a lot of nonsense built in, but unless you go looking for it you will probably not become aware of it. It also can be complex to set up correctly like you mention, but that sort of complexity bothers me much less. That's more of a "once and done" thing, which I find much more tolerable.
My tolerance for configuring software is actually very nearly pegged at zero, unless I believe that the configuration I am creating will still be useful for at least 10 years. This rules out configuring just about anything for me, with very few exceptions. My current significant configurations are for vim (the emergence of neovim gives me some concern) and zsh (aliases and basic keybindings only. I am confident that any shell I'll be using in the future will allow aliases). My window manager, browsers, etc all use their default configurations (I got burned by Awesome WM changing their configuration scheme a few years ago).
Configuring Eclipse to be tolerable is probably possible, but I have zero confidence that any such configuration could last even two or three years.