That said, vim and emacs can be in a way a component of a DIY IDE - you just need to put the work in and build it yourself. I exclusively use vim with a handful of useful plugins, scripts, and additional CLI tools in other terminal splits.
The terminal is my IDE.
(Of course if you're writing programs in Elisp, Emacs is a more tightly integrated IDE, but the success of Emacs, org-mode, Magit, and Amazon notwithstanding, Elisp is not a very good programming language. Also, if you run Emacs in a terminal, you can quite reasonably use it over a low-bandwidth, high-latency ssh -C connection, and it becomes good for remote pair programming, which is not true of running Emacs in VNC. I haven't tried running it in Xpra.)
Historically Emacs started up far too slowly and used far too much RAM to be convenient to use it in the way you're describing, as a text editor invoked by a larger IDE; I remember waiting over a minute for a new Emacs process to start up in the mid-90s. Nowadays it might be reasonable, but it still takes almost 4 seconds on this laptop, which I find to be a painfully long wait for opening a file. Also, Emacs's language-agnostic autocomplete M-/ works a lot better if you have a lot of files open in the same Emacs session.
Although there are packages that configure Vim to work the same way, it is more common to use Vim in the way you describe, as a component of the IDE rather than an IDE in its own right. And Vim has the great advantage that it starts up instantly, and the advantage or disadvantage that its equivalent of M-/ (^P and ^N) is buffer-scoped, so it doesn't matter if you have more files open.
That is one of the use cases for emacsclient/Emacs server.