Hacker News Reader

I don't understand why you'd want a tokenizer to be fast...

Surely the output of said tokenizer you're going to stick through a machine learning model that is orders of magnitude slower?

So it really doesn't matter if the tokenizer takes microseconds or milliseconds, when the main model takes seconds for the same input/output.

View on HN

Surely there can't be a microsecond to second ratio between the two steps, because then they would never be able to crawl the hundreds of billions of documents the model is trained on. Once the model is built, sure, the front end is irrelevant.

The ratio between the tokenizer and the model is constant for both training and inference.

And I believe the ratio is that big - you just use a lot of compute for training, and that's why it's so expensive.

Inference is latency sensitive so the front end is still relevant.

Reply on news.ycombinator.com