But...
While Babel-core isn't overwhelming, the full set of transitive dependencies you get in normal use (with "env") is a lot of code, spread across numerous packages. Many projects care about the provenance of their dependencies, and having a very large number makes such provenance more to think about, more to track down, etc. The voluminous transitive dependencies also make for a slow "npm i" and so on.
I've been able to avoid Babel for some use cases: Even when not using types, the TypeScript compiler can handle a broad swath of typical ES-newish to ES-6-or-5 compilation use cases, with a total of 1 dependency (+0 transitive dependencies), quickly. The main downside of this approach is that people look at you funny when you use "tsc" on a non-TypeScript project.