What Are Tokens in LLMs? | Hacker News Reader

What Are Tokens in LLMs?

bearisland.dev

10 pointss1monb6 comments

The article comes from the "personal" experience of an LLM so it's a very trusted source!

> This isn’t because the model can’t count. It’s because it never sees the letters at all.

> The chunks aren’t characters and they aren’t words. They’re something more specific, and the specificity matters more than most people realize.

> Those numbers are real, but they hide what a token actually is.

> GPT-4’s vocabulary isn’t Claude’s. Claude’s isn’t Llama’s.

> The model never sees text. It sees a sequence of integer indices into its own private alphabet.

> So tokens aren’t “roughly like words” or “kind of like characters”. They’re the atoms of perception for one specific model, and they’re the only language that model speaks.

> The same sentence is nine tokens to GPT-4 and seven tokens to Llama 3. Not because Llama is smarter or the sentence changed, but because the two models have different vocabularies.

> That’s it. No clever scoring, no neural network.

Could people who use LLM to write articles at least prompt them to have a better style? I'm really tired of the default Claude style (a lot of Chinese models also reuse the same style)

I appreciate the feedback. My main focus was on the visual elements, and not so much "ridding the text of AI-traces".

What did you think about the more visual elements?

Simon

I will do better and link to the research and related sources in the next iteration.

I was just pointing out how the article is clearly LLM written, probably including the interactive widgets. It's especially obvious because someone writing such an article in 2026 would at least find what the newest tokenizers are, instead of mentioning LLaMA 2/3 (!), and GPT's old tokenizer that they dropped since GPT-4o (or something close).

And, more obviously, the fact that GPT-4 is being directly named even though that model is over 3 years old by now: "Ask GPT-4, Claude, or Gemini today and they will usually answer three.".

Sorry, I just think that the article wasn't produced by a human at all.

> It's especially obvious because someone writing such an article in 2026 would at least find what what the newest tokenizers are

The underlying BPE algorithm, which is the main focus of this article, is the one used modern tokenizers today.

> The fact that GPT-4 is being directly named even though that model is over 3 years old by now

That is fair. Will be updated

> Sorry, I just think that the article wasn't produced by a human at all.

While I have used LLM to help me write and explain my content, my hopes is that most readers does not share this opinion of yours. Everything touched by AI is not slop, and I wanted to share the notes I created for myself.

Reply on news.ycombinator.com