Modern IDEs are following a platform + plugins architecture that is probably as flexible but more robust than whatever you can tape together around vim.
It's not that using an IDE with a vim mode is not a possible viable alternative, but it's a game of trade-offs, as usual.
I really hope in the future I'll find the time to do that, but for now I am content. It's always a trade-off.
Vim also starts up without any noticable delay and can be used for every text file, no matter in what context.
Only if you run it without any plugins. Unfortunately if you have a lot of third party packages, vim startup times start to get closer to emacs startup times (both of which are still better than any IDE I have used, of course). I keep a vim.norc alias in my alias file which launches vim with an empty .vimrc for this very reason.
I’m not very experienced with complex plugins, but most of what I’ve seen did nothing unless explicitly initiated. Maybe there are global variables to control that.