If you're building a large or even moderately sized app, your layers of in-house JS will still be a framework.. and less optimized than one that has had all of the engineering effort put into it already.
The benefits of frameworks are overrated. Layers of abstraction may ease certain tasks, but also require additional learning (the language and the framework). Worse is abstraction on top of abstraction! This is a natural result of the midset that software must always be added to and built upon. Not only do we build the layers on top of layers with the tools. We build layers on top of layers IN THE TOOLS.