In my case, I wanted to build a (rough) superset of JS, adding the syntax and language features I was interested in but otherwise starting from a working (tested!) language. This made the task dramatically easier.
Forking babylon, Babel's parser, worked well for this in my case, but you could also write your own parser from scratch and use babel-types to build your AST.
In my case, I have a ~1,000-LOC babel plugin "compiler"[0] with maybe 1,000~2,000 LOC of mods to the parser[1], resulting in a language that looks fairly different from JS but feels pretty solid. Users of the language benefit from Babel's tooling – it works with webpack, ESLint (decently), etc.
[0] https://github.com/lightscript/babel-plugin-lightscript/