Hacker News Reader

Comment by karimf | Hacker News Reader

Yes, and also waiting for the next iteration of Gemma. Muse or Qwen are optimized for coding, while IMO Gemma is still better for non-coding tasks.

https://x.com/osanseviero/status/2086107547535122767

View on HN

I am working on a project where we have to classify customer calls into more than 10 categories. As the client wants everything locally I tried a few local LLMs. Gemma turned out to be the best model for this task. The classification accuracy is impressive, and the client is happy that I am using an American model.

I guess the client is American.

Not necessarily. Except for SpaceXAI's (haha) models, there's less risk of reality distortion in Western models. Probably doesn't affect his use case but you wouldn't really want to ask Kimi K3 for a history of China for example.

You can partially tell by the tokeniser; which gives you some hint into the training corpus mix.

</div> is four Gemma4 tokens, but one Qwen3.6 token.

Looks like we have a /r/localllama dweller here.

Where do you find this information for each model?

When you look on HuggingFace.co at the files of a model, for each model you will see a file "tokenizer.json".

In that file you can see all tokens and their corresponding numeric codes.

The tokenizers are included in the open s̶o̶u̶r̶c̶e̶ weights releases; you wouldn’t be able to use the weights without the corresponding encoder/decoder, in fact.

Reply on news.ycombinator.com