Hacker News Reader

Comment by Tiberium | Hacker News Reader

Yeah, Anthropic's current tokenizer in Sonnet 5/Opus 4.8/Fable 5 is much worse than OpenAI's. Also, OpenAI has been using their current o200k_base from the day GPT-4o came out over two years ago. Just a few of my own tests:

- A ~2000-2002 legacy C++ game codebase at about ~90kloc: GPT 1.12M, Claude 2.2M

- A ~30kloc TypeScript codebase: GPT 260K, Claude 437K

In the end, GPT's current tokenizer is ~1.6x-2x better than Claude's current one, depending on your data. And you can check for free for both, for OpenAI just use the open-source libraries, for Anthropic - you have to use their count_tokens endpoint as they don't publish the tokenizer, but the endpoint is free (and allows requests over 1M tokens as well).

View on HN

Interesting... Naively I'd assume you'd have a pretty unfair advantage on quality if you have materially more information dense tokens.

That doesn't really appear to be the case as GPT and Anthropic models appear evenly matched despite Anthropic encoding the same text into almost ~2x the tokens...

I'd also - naively - assume this would make training their models more expensive. Though inference now dominates, and they'd probably rather have more tokens than less (to charge you for them at future 80% margins).

If a given paragraph gets encoded into twice as many tokens, that means the model gets twice as many matmuls to process it. The amount of compute thrown at the problem is increased (everything else constant), which may improve the quality of the result. This is believed to be one of the reasons that 'thinking' tokens improve quality. For long tasks it will lead to more context compactions though which will harm the quality to some degree as well.

It would be nice if inference could somehow perform token generation using "contractions" of "fluffy" tokens, where combining those tokens doesn't decrease nuance but provides additional efficiency. That may already be happening - I haven't looked at the most modern methods of inference in a long long time.

more dense tokens means more stuff to fit into the embedding space which is per token, so more work to disentangle later

you use the wrong word

the Anthropic tokenizer is not worse, its more expensive/verbose

So, worse? Because we benchmark off token use when talking about token use, and everyone else understood that.

The most important feature of a tokenizer is dividing the inputs into independent values that the neural network can work with. It's not the size.

I mean it might lead to better performance on the model side. So the tokenizer is better but more expensive.

It’s better for them

Reply on news.ycombinator.com