Three reasons spring to mind:

1- the DOM APIs are very verbose, excepting perhaps replacing chunks by setting innerHTML, but

2- manual DOM manipulations aren't composable without also using web components

3- naive manual DOM manipulation can potentially be slow by way of triggering needless layouts/repaints or inefficient use of the APIs

Web components didn't exist when the major frameworks came about, and the ones before them handled composition rather poorly if at all (such as backbone). Angular, ember, react and company all sought to provide a cohesive story for code reuse and composition. Many had their own performance stories, though it was always a secondary concern to everything else.

I've done a bit of everything, including pure vanilla JS, just jQuery, vanilla + web components, all the way up to next, ember, angularjs and angular, and more. If you're building a web site, vanilla JS is fine. If you're building a web "app", you'll pretty quickly appreciate what a framework can bring to the table. If you don't, you'll also quickly find yourself building your own mini framework and runtime to manage things.