First, whatever you don't use, it's not even loaded in memory.
Second, bloated is all about having tons of options you don't need or use. Not about adding stuff you DO need piecemeal.
Third, bloat is mostly a UI thing, not a "number of add-ons" or "too many lines of code" thing. Programs don't get slow because they are "bloated" with extra code (if it doesn't run, then it has 0 effect on their speed). They get slow because they are badly programmed (e.g. loading one big text file all at once in memory instead of having a paging system).
The availability of tens of thousands of packages hasn't made Emacs "bloated", much less Vim or ST. Or even installing those packages doesn't make those editors feel bloated.
Whereas something like Eclipse was bloated from the start -- because it was a very heavy design with tons of abstractions layers, ton of built-in options and visual clutter etc, created on a GCed language with frequent stops on larger codebases etc. That's even without any third party plugin added, just the Eclipse Java SE core packages.