The standard way to use Emacs is as an IDE; you use M-x compile to run the compiler in a window under Emacs and M-x gdb to run the debugger in a window under Emacs, in both cases with hypertext capabilities. These are standard packages included in a default Emacs install, although you may want to bind them to more convenient keys. (I bind [f5] to recompile.) You do not need to run Emacs in a terminal, and copy and paste to and from Emacs (not to mention things like viewing SVGs you're editing) works better if you run it in a window on its own. You do not need to "put the work in and build [an IDE] yourself" if you are using Emacs.

(Of course if you're writing programs in Elisp, Emacs is a more tightly integrated IDE, but the success of Emacs, org-mode, Magit, and Amazon notwithstanding, Elisp is not a very good programming language. Also, if you run Emacs in a terminal, you can quite reasonably use it over a low-bandwidth, high-latency ssh -C connection, and it becomes good for remote pair programming, which is not true of running Emacs in VNC. I haven't tried running it in Xpra.)

Historically Emacs started up far too slowly and used far too much RAM to be convenient to use it in the way you're describing, as a text editor invoked by a larger IDE; I remember waiting over a minute for a new Emacs process to start up in the mid-90s. Nowadays it might be reasonable, but it still takes almost 4 seconds on this laptop, which I find to be a painfully long wait for opening a file. Also, Emacs's language-agnostic autocomplete M-/ works a lot better if you have a lot of files open in the same Emacs session.

Although there are packages that configure Vim to work the same way, it is more common to use Vim in the way you describe, as a component of the IDE rather than an IDE in its own right. And Vim has the great advantage that it starts up instantly, and the advantage or disadvantage that its equivalent of M-/ (^P and ^N) is buffer-scoped, so it doesn't matter if you have more files open.