curioussquirrel

As usual, take it with a grain of salt.

My first real programming achievement was building a website for an Ultima Online shard. I wrote some really terrible PHP and HTML, but it worked for 20+ years afterwards. Great memories!

I was surprised that there is still an active community around UO!

In any case, this is very cool. Thanks for sharing!

V4 is definitely a step-up from V3.2 on our multilingual benchmarks.

Two caveats: - when inferring through Openrouter, we've had a lot of issues with very slow speeds (TPS) and an occasional instability. I just checked and it's still 10-30 TPS on all available providers, which is not a lot for a model that likes to think as much as DeepSeek does.

- the official DeepSeek API makes no guarantees of data privacy even for paying users.

Both points could be moot with using it through Azure AI foundry (the latter is, afaik); I have yet to test that.

In any case, happy to see more open-weights models that are somewhat competitive with SOTA models!

Came here to say the same. Please add a few screenshots!

MoE is mostly an optimization of the active parameters and therefore lowering the compute requirements, but it can provide some performance improvements over dense models in some cases.

I would not describe reasoning as optimization: In fact, it's typically doing the opposite, as models spend way more tokens (and therefore compute) on responding to the same prompt. Some of the smartest models these days use ridiculous amounts of reasoning before they ever respond. Try Deep Research in Gemini or Claude and you'll see what I'm talking about.

>> But this seems to be a very clear path to be "taking the car to the carwash by foot" for a long time, isn't it?

I thought the progress was plateauing sometime last year too, but then some new models got released and we saw that the multilingual capabilities improvements are real. And if you want something more tangible and reported on, consider the Opus 4.5/4.6 coding revolution (Claude Code explosion) a few months back.

LLMs being stochastic and statistical machines, there will always be funny things that people will come up with that will trick them, be it R's in strawberry or the carwash by foot. At the same time, I can tell you from my experience that a lot of the Misguided attention ( https://github.com/cpldcpu/MisguidedAttention ) type of stump questions work at a much lower rate with newer models. Progress is being made, it's just not in visible areas.

BTW, you can come up with many trick questions that will stump even humans with PhDs. They will be of different kind than the ones for LLMs, but this is not a flaw unique to LLMs.

If you're asking whether the progress to AGI isn't taking too long, then I personally think LLMs, at least with their current architecture, are not the foundation of AGI, and will always have inherent limitations. But we're fully in the "that's just like, your opinion, man" territory now :)

There are architectural changes (such as reasoning or mixture of experts) that measurably improve how well models perform. So the improvements are definitely not just from data.

I can speak for my area of expertise - multilingual capabilities. Some SOTA models are making huge strides in their support of various languages, and increasingly they understand and can produce text in languages where GPT-4 era models were absolutely lost. These are probably from a combination of richer training dataset and architectural improvements (more parameters?).

I posted about this here if you're interested: https://news.ycombinator.com/item?id=47847282

Now that doesn't necessarily mean that models are also getting substantially better at English or other major languages. They likely are to some degree, but we've reached a point with major languages where core linguistic proficiencies are covered, and what's left is the more squishy part: style, tone of voice, ability to use different registers naturally, or what some people would call linguistic taste. But that's much harder to measure and therefore trickier to provide evidence for.

Hope this helps.

Edit: typo, clarification

Here you go! https://news.ycombinator.com/item?id=47847282

Just saw your thinking edit! That's a great question and one I wanted to study in depth, but these days you don't really get access to the raw thinking data. It's usually summarized and you can't even be sure what language the model thought in unless you have access to the logits (so only viable for open-weights models).

I am fairly convinced that there's a certain polyglot snowball effect: once the LLM is fluent in 20 languages, it can pick up on similarities in vocabulary, syntax etc. and learn the 21st language with much less effort (and training data). This might be difficult to actually study in an isolated way, but it's a real effect for humans and it makes sense the the pattern matchers that LLMs are would find these shortcuts.

Using similar words should land you in similar places in the latent space, even if they actual word or their order is slightly different. Where it gets interesting is how well English words map to their counterparts in other languages, and what practical differences it makes. From various studies, it seems that the gravitational pull of English language/culture training data is substantial, but an LLM can switch cultures and values when prompted in different languages.

One more thing: we're working on a multilingual benchmark that will evaluate core linguistic proficiency in 30 languages. We already have a lot of data internally and I can tell you that:

- Gemini 3 Pro is a multilingual monster.

- GPT-5.4 is a really good translation model, big improvements over previous subversions in the 5 family.

- Opus 4.6 is good but usually third place.

- Somehow, Grok 4.20 is surprisingly good at some long-tail languages? Its performance profile is really odd. Unlike all the other models.

EDIT: layout

Disclosure: I work at RWS/TrainAI, we did this study. Recently I alluded to it in a comment and was encouraged to share it, so here it is! We focus on multilingual proficiency, which tends to be understudied: most benchmarks are English-heavy or even English-only and don't tell you much about how models actually perform across languages. This is our second iteration of the study. 120 linguists, 8 models, 8 languages, 4 tasks, every output blind-reviewed by 3 native speakers.

Some notable insights:

- GPT-5 is strong at text normalization and translation but regressed on content generation vs GPT-4o. Chinese outputs had spacing/punctuation issues, Polish read like "translationese" even with no source text.

- Gemini 2.5 Pro scored 4.56/5 on Kinyarwanda. In our first study (late 2024), no model could produce coherent text in that language.

- Top LLMs outscored humans working under realistic constraints (time-limited, single pass, no QA). Humans didn't rank 1st in any language. (We're now planning a follow-up to zoom in on that.)

- Tokenizer efficiency matters again: reasoning models burn 5-10x more tokens thinking. Claude Sonnet 4.5 encodes Tamil at 1.19 chars/token vs Gemini's 4.24 — ~3.5x cost difference for the same output. There has been a lot of talk about the Opus 4.7 tokenizer, this is the same issue, just in multilingual setting.

If you find the study useful and want to help us convince the execs to keep funding this, a signup on the landing page goes a long way: https://www.rws.com/artificial-intelligence/train-ai-data-se...

Happy to answer questions!

Yes, but post training cannot possibly account for all possible use cases. Sane defaults are fine, you can't really do much about sampling parameters in chatbots and coding harnesses anyway. And when making an API call, you have to actively change the parameter in your payload. I don't believe there's any real risk.

After Anthropic, Moonshot is another model provider who restricts tweaking of sampling parameters. I do like the idea of the vendor verifier, though.

There's been quite a few threads about Opus 4.7 but none of them seems to have discussed some breaking changes on the API side, particularly removal of sampling parameters.

From the migration guide: >> Sampling parameters removed: Setting temperature, top_p, or top_k to any non-default value on Claude Opus 4.7 returns a 400 error.

Let's set aside that this should probably be a deprecation warning and not a 400. Not having these dials limits utility for cases like synthetic data generation, natural language QA and many more. Even though temp=0 does not guarantee determinism, getting 99 identical responses out of 100 is reasonably close to determinism for most practical use case. Default temp gives you wild swings in performance which temp=0 almost perfectly eliminates. And there are valid use cases for using temp=0 or experimenting with different values.

The writing was on the wall since even earlier Opus versions would override temperature setting and reset to default when thinking was enabled. Now there is no way to control it at all. It is a bit disappointing.

I understand most people around here will be using Opus in Claude Code or in another harness for coding, and in that case you are not really affected. But for those of you building products and using the API in different ways, how are you dealing with this change?

If anyone from Anthropic is reading this, any insights into why it was removed would be great. I am struggling to believe this is because of distillation concerns. Thanks!

Will do! Thanks for the encouragement

Claude's tokenizers have actually been getting less efficient over the years (I think we're at the third iteration at the least since Sonnet 3.5). And if you prompt the LLM in a language other than English, or if your users prompt it or generate content in other languages, the costs go higher even more. And I mean hundreds of percent more for languages with complex scripts like Tamil or Japanese. If you're interested in the research we did comparing tokenizers of several SOTA models in multiple languages, just hit me up.

Thanks for sharing! Have been begrudgingly using Darktable since that seems to be your best option on Linux, but the UI/UX never really clicked with me. I wish this was opensource but I will give this a shot (pun intended) for sure.

Thank you for the transparency and insights! Very helpful.

We actually did the same thing re generating charts in brand style to avoid any mishaps, since then I sleep much better

Absolutely unhinged and very entertaining. Thanks for sharing!

Give Gemma 31B a shot for translation, it does a very good job at that given its size.

We're doing multilingual testing and I can confirm what you've observed: Gemma 4 is surprisingly good at multilingual tasks, especially given its size. This is mostly true for the dense 31B model.

Same, I quickly tested it for code gen and it produced mostly good code for simple problems, but it sometimes hallucinated words in non-English scripts inside the code.

For anyone interested in multilingual performance, which is not usually well benchmarked or reported: Gemma 4 does really well, especially the dense 31B version. In fact, it outperforms many models with an order of magnitude higher number of parameters.

It is not quite capable of performing work on really long tail languages, but their claim of 35 languages supported (and a hint of some knowledge of up to 140) was substantiated by our tests.

If you're doing work outside of English and/or need to run a translation model in your terms, Gemma 4 is a very good candidate.

Thank you. +1. There are obviously differences and things getting lost or slightly misaligned in the latent space, and these do cause degradation in reasoning quality, but the decline is very small in high resource languages.

Such a good game and execution. Thank you

And regarding specific models - we obviously only tested a few languages, and there are thousands of them in the world. But Gemini seems to lead the pack basically regardless of the language your throw at it. YMMV.

In our previous tests, when it was 1.5 Pro against GPT 4o and Claude Sonnet 3.7, Gemini wasn't winning in the multilingual race, but it was definitely competitive. 2.5 and 3.0 seems to be big leaps from the 1.5 days. That said, it also depends on the testing methodology; we tested a bunch of use cases mostly to test core linguistic proficiency. Not as much complex tasks in language or cultural knowledge.

This is probably not a core concern for most HN readers, but at work we do multilingual testing for synthetic text data generation and natural language processing. Emphasis on multilingual. Gemini has made some serious leaps from 1.5 to 2.5 and now 3.0, and is actually proficient in languages that other models can only dream of. On the other hand, GPT-5 has a really mixed performance in a lot of categories.

You could ask GPT for what it knows about you and use it to seed your personal preferences to a new model/app. Not perfect and probably quite lossy, but likely much better than starting from scratch.

Animation generator without a single demo/example animation on the page?