javascript used to be awful. Using JQuery was more or less required to have a decent API and not have your entire app full of polyfills. Those days are behind us, but the precedent remains.
Some javascript APIs are still pretty bad. Fetch and XMLHttpRequest are both really warty in their own ways, to the point where you're probably writing a wrapper function. Common-enough collection operations are missing, you'll need to write some sort of utils file. So you're definitely going to want at least some sort of small library for the missing stuff.
The DOM forms a parallel, persistent tree to your application state, and you need to manage the relationship between that and your app. Parallel state trees just always suck (I've also dealt with them in Box2D, some game engines, SVG renderers...), it's probably an antipattern. It's not that big of a deal for a small app, but for larger ones it starts getting hairy. And it certainly feels like you should be able to write a better API easily enough, but because the problem is that the entire DOM is based on a parallel state tree model, your wrapper library ends up getting large before you know it.
There are good technical reasons to use some sort of build system for your web content. Concatenating and minifying all your javascript helps a ton. But to do a really good job of minifying you basically need a whole compiler, and at that point you might start pulling in all sorts of clever transformations that seem like a good idea.