Hacker News Reader

The llama 3.0, 3.1, & 3.2 all use the TikToken tokenizer which is the open source openai tokenizer.

GP is talking about context windows, not the number of token used by the tokenizer.

Somewhat confusingly, it appears the tokenizer vocabulary as well as the context length are both 128k tokens!

Yup, that's why I wanted to clarify things.