Rebuilding Babel: The Tokenizer

nan.fyi

36 points·pgayed··5 comments

FWIW, a tokenizer is not really necessary. It (arguably) makes things cleaner / easier, but you can also just write a parser that operates directly on a byte stream, if you feel so inclined, and it turns out fine.

Yes I agree. The article actually says the purpose is

> To separate the logic that handles a language's microsyntax from the logic that handles regular syntax.

But everything is just syntax. This distinction between microsyntax and regular syntax is really just for convenience; and sometimes it's not that much convenience.

And isn’t this developer convenience at the cost of performance in a piece of software where performance is very important?

Today as my “end of the week fun experiment” I replaced Babel with esbuild on a years old 50k sloc production web application. It took 30 mins and builds now take 500ms instead of 63,000. Not quite apples to apples yet. But it’s such a compelling change that I can’t ignore it. I’m excited to see how Babel steps it up in the years ahead.

Shouldn't be any noticable perf impact of splitting out tokenization as a separate stage, I think.

This is way more in depth of an evaluator than I've ever written, but it's a fun project. I'm a largely self taught dev and build an evaluator was an illuminating project for me. Mine was just a math parser but this is a cool project!

Reply on news.ycombinator.com