Ehh... the reductio of this argument is writing everything in assembler (libc? giant hunk of third party code right there). I surmise that, by comparison, the blowback you encountered was relatively minor.
Because most of the complexity comes from code (esp. libraries and drivers) trying to solve a larger problem than you actually have.
That's the same reason why, when you follow that logic, you eventually write your own Forth. Not because it fun. Not because you want to learn about Forth or compiler. But because my Forth solves my problems the way I see fit, her Forth solves problems the way she wants, and your Forth is going to solve the way you want.
The difference I'm drawing being, libraries just provide a mess of utility functions. Theoretically, even if your compiler won't strip the library stuff you don't need, you'd be able to take just the bits you need by copy/pasting a relatively small volume of code. And dropping the library would be a small change, that just requires finding replacements for the functions and classes you were using.
Frameworks tend to involve some Grand Unifying Abstraction that you need to inherit from, and that gets imposed on your own code. Things tend to be so tangled together at a conceptual level that it's not really possible to use them in an a la carte manner. Migrating off of a framework tends to require more-or-less a rewrite of all the code that interacts with it.
To take some Web examples: jQuery's more on the library side of things. D3 is more of a framework. React is very much a framework.
"High level" means details abstracted away and solved so you don't have to think about them. Our CPUs understand only the most primitive of instructions; the purpose of all software is to climb the ladder of abstraction, from a multiplication routine abstracting over repeated addition, to "Alexa, set an alarm for 8 AM." To write things yourself is the very essence of descending to a lower level.
Abstraction comes at the price of loss of fidelity, yes - Alexa might not ask you to specify exactly what form your alarm will take - but the benefits are a vastly increased power/effort ratio. It's worth it, because most of the time you don't care exactly how a task is done - you just care that it IS done. And - mostly - your needs are not that special.
Frankly, sharing information on how to do things so that others can build upon them is the only reason we have technology at all. Perhaps you've read "I, Pencil"? With a lifetime of effort and study, you would struggle to create a single pencil drawing from "scratch". Chuck Moore's supposedly astonishing productivity notwithstanding, I notice that all of the software I actually use is a heavily layered tower of abstraction (and, curiously, none of it is written by Chuck Moore). It appears that by and large the choice is between layered, multi-author code - and no code at all.
Perhaps you never saw the images from the Philae space probe? Because that's an RTX2010 that powers it, one of Chuck Moore's designs.
Maybe you don't use Moore's software directly, but you never know when it has been used for you [1].
[1] https://wiki.forth-ev.de/doku.php/events:ef2018:forth-in-tha...
See also: Emacs, the ultimate customizer's editor, easily shapeable to your particular needs -- and currently losing badly to Visual Studio Code which is only readily customized with configuration options and third-party packages. When you need to pair or mob, having a common toolset and vocabulary beats having a special-snowflake environment.