This is similar to my own experiences.
During the past decade the JavaScript ecosystem had to evolve and adapt to handle increasingly more advanced requirements. Multiple tools and libraries rose up to meet the challenge, and the best ones have slowly floated to the top. It's not like the winning tools are even that recent: I think Webpack was already around before 2012, TypeScript was released in 2012, and, if I remember correctly, React was released around 2013. Fun fact: babel was called 6to5 when it was initially released.
Webpack can sometimes be difficult to configure because the problems which it sets out to solve are also complicated. To be fair, their old docs were quite lackluster, and the creator's coding style was uncommon, which made reading through the source a bit challenging. I know a lot of people are critical of Webpack, but I was overjoyed when I first discovered it, as it enabled me to correctly solve a huge number of issues. Once adopted, it let me drastically simplify our build pipeline, as well as our development and testing environments.
I can sympathize with non-frontend developers because I also feel overwhelmed sometimes when I look at all the tooling related to backend / devops / infrastructure stuff. But I understand people build and use these tools because they're usually solving fairly complex real-world problems. I suspect that many people that complain about JavaScript don't care much about the frontend and they just want to get it out of the way as quickly as possible. They want to pick up whatever standard tools the community has centered around and hammer together something without having to dive in too deep. If the developer looks down on frontend development and they encounter a difficult problem which they're unable to solve, then they must deal with the cognitive dissonance.
To give a concrete example of something I've struggle with: I can barely make sense of the containers ecosystem. Using docker-compose during development to bring up an application's external dependencies is amazing. However, I generally haven't had much success configuring and using docker to run an app locally. There's always performance issues, or I can't get something to work correctly, or I can't figure out how to get a good workflow going. Since we're on the topic: if anyone knows a practical example of a regular web app (e.g. rails or node, but I'm not picky) which has a few workers and which depends on a few services (e.g. Postgres, RabbitMQ, Redis, MinIO), I'd love to check it out. Oh, and I haven't even mentioned deployments and Kubernetes, which seems like a whole nother dragon.
If you're not a fan of frontend development I'd suggest using server-side rendering with occasional scripts dropped in for extra functionality. Think carefully about your application and its requirements; not everything needs to be a Single Page App. Depending on the kind of application you're building, you might want to evaluate next.js [0] and create-react-app [1].
[0] https://nextjs.org/
[1] https://github.com/facebook/create-react-app