You can't build anything without Babel for the most part since ES6 code needs it to transpile down to ES5.
The end result is JavaScript's entire ecosystem is a fragile mess.
You can even use ES6 modules directly in the browser, including import/export syntax. When I measured back in 2018 100% of our users of the company I was I was in had support for that. With brotli and HTTP2 the performance is comparable to bundled code.
The only holdover is JSX, of course, and maybe some things like the ?? operator that don't work in the client-side yet. But Typescript can take care of that. You can even tell Typescript to target ES6 so you can use ES6 modules in the browser, and this will let you eschew Webpack.
So no, you don't really need Babel in 2020. :)
If you are using Babel as a compiler, you can enable the TS syntax plugin to have Babel strip TS type syntax out of its processing output, although this does not do any typechecking during the compilation process.
Any chance of JS being mostly Babel free by next year ?
Every major web framework's standard build tools (Create-React-App, Vue-CLI, etc), are based on Webpack + Babel, and most folks who are using one of those frameworks but setting up their own build toolchains still base them around Webpack + Babel.
There's some potentially interesting technical alternatives out there, but Babel is a keystone in the modern JS ecosystem.
It would also detect some probable bugs in your js code.
Of course after a day or two you would start writing typescript and never look back.