gertlabs

GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing.

In multi-agent coding environments, GLM 5.2 is just shy of Opus 4.6 on average. Data at https://gertlabs.com/rankings

But when factoring in performance/cost, GLM 5.2 is the frontier model.

Qwen 3.6 27B is an anomalously strong all-around model for its size, but when we run our evaluations, we generate 10 coding submissions/language/model (110 total). So full discosure, the per-language per-model performances can be noisy (I do not think Qwen3.6 27B is better than Fable 5 in agentic workflows when writing Kotlin, given enough samples, although we do find some interesting anomalies that hold up under large sample sizes).

By domain, I really meant "tool calling" and "one-shot fluid intelligence"

Anthropic models were the original leaders in tool calling and agentic work, even when other models felt significantly smarter in (Claude Sonnet 3.5 vs Gemini 2.5 Pro, for example). OpenAI models were the opposite, starting smart (more correct solutions on the first try) and got better at exploring and iterating with tools in 2026. The latest releases (Opus 4.5+ and GPT 5.4+) excel at both.

One-shot performance often translates to the most difficult problems a model will be able to understand. We run an evaluation that tests both agentic and one-shot performance, and we find that Chinese models are almost universally very good at using tools and a harness to iterate towards a better solution, whereas their initial response ranks relatively low.

Compare that to Gemini models, which have impressive fluid intelligence on the first response, but fail to call tools or explore correctly which limits their usefulness for agentic coding.

Neither will be great for coding in a computational chemistry repo for different reasons, but the model with strong one-shot performance will be less likely to make subtle errors indicative of poor understanding, so we weight both capabilities into their final score.

The latest Anthropic and OpenAI models excel in both domains.

Data at https://gertlabs.com/rankings

They're within confidence intervals of each other, but remember how much discussion there was that Opus 4.6 had been nerfed in March. We averaged samples over the entire lifetime of Opus 4.6, which likely served many different underlying checkpoints. Even the best version of Opus 4.6 was hardly an upgrade.

We find a lot of interesting anomalies with our benchmark that hold up under large sample sizes.

All of our posts have been well received by an insanely high percentage of people who have interacted on here -- most people clearly find what we're doing interesting and relevant to the HN community (AI evaluations). A flag seems pretty aggressive! Especially when the top comment on the article (after our above comment got flagged) is about tacos.

I'm a person running the account, and I only post where I think we have a relevant contribution.

On our multi-agent coding and reasoning evaluations, GLM 5.2 is the first model we've tested that crossed the threshold of being on par with or better than Opus 4.6 (although as usual, we have GLM 5.2 and most other Chinese models a bit below most other benchmarks with test methodologies that are more vulnerable to benchmaxxing).

Data at https://gertlabs.com/rankings

GLM 5.2 is the first model we've tested that is unambiguously on par with, or better than Opus 4.6 (although as usual, we have GLM 5.2 and most other Chinese models a bit below most other benchmarks with more vulnerable test methodologies).

Data at https://gertlabs.com/rankings

It's likely overfit to common harnesses and iteration patterns, so it struggles with formatting tool calls and json in our testing which use our own harnesses (although there is a lot of overlap with tools that would be found in any coding harness like bash, apply_patch, etc.)

We didn't love the results because it draws negative scrutiny to our benchmark, but the results are real and done at scale and I think DeepSeek V4 Pro's inability to do agentic work outside of environments it was trained on is an important thing to measure, especially when so many other models can generalize to new environments just fine.

Google models also struggle with tools, but they have very strong initial answers, so there is more potential for them to bridge the gap with some better post-training.

DeepSeek v4 Pro struggles with a custom harness, and all the models ranked above it don't, so it gets downweighted in the agentic coding benchmarks (although it ranks better than Flash in one-shot problem solving: https://gertlabs.com/rankings?ow=1&mode=oneshot_coding). We ran plenty of samples.

MiMo v2.5 is on there, as well as the pro version.

We found a few anomalies in our evaluations, which makes sense -- if every new sub-release is better across the board in every area of the model card, that should raise alarms about benchmaxxing. But the main thing we found is that hype != performance, and I trust our benchmark methodology significantly more than the model cards the labs add to their press releases.

MiMo V2.5 Pro (regular speed) remains the strongest open weights agentic coding model we've tested -- it's been interesting to see how little attention it has received relative to some lower performing releases. And the "fast mode" pricing is very competitive here.

Data at https://gertlabs.com/rankings

GPT 5.4+ models are extremely good at writing Clojure, agreed. In the agentic coding part of our benchmark, they do have access to the REPL via bash if they choose to use it. Filtered here: https://gertlabs.com/rankings?mode=agentic_coding

I might just be a simpleton -- I never had the resolve to try an ambitious project in Clojure. I was not aware that you could get full OOP though, what you are describing feels like yes technically possible but kind of a hack to get inheritance / no type hierarchy enforcement. I'm no expert on the language though

Success rate includes syntax/compilation failures as well as environment rule violations, and is almost entirely from one-shot code generations. Percentile shows how well the working submissions perform.

In long horizon agentic coding evaluations, strong models fix the syntax and percentile and it becomes a direct comparison of which submissions per language performed the best on average. You can filter for that here: https://gertlabs.com/rankings?provider=openai&mode=agentic_c...

The functional paradigm is a bit uncomfortable at first, but it does make problem solving feel... different. I personally find OOP to be the most intuitive for large scale systems design, but that's just me.

Most models do not perform particularly well in Clojure, but OpenAI models fully utilize the power of the language. Subjectively, it kind of seems to match the personality. Data at https://gertlabs.com/rankings?provider=openai

Nice, that's a good one -- interesting dynamics can come out of deceptively simple social games.

OpenRouter is our primary provider for evaluation data, and we've been really happy with them!

I'm sure they're experiencing growing pains, but a larger model selection (and faster releases for open weights models), would keep us from using other providers. For example, it took much longer than it should have to get Qwen 3.6 ~30B class models released (almost 2 weeks if I recall)

That's an interesting point -- they are told that they are paper trading. Maybe we should run another session that A/B tests this.

We're a month into a long running experiment to see how recent models perform in day trading, where they have a constant harness giving them the ability to write code, access the web, take notes, and install handlers to trade for them. Realistic slippage and fees and margin requirements are built into the simulation.

The harness and the capabilities built into the environment gives the models all the resources they would realistically ever have access to in order to help them succeed (or fail).

We just tell the models to maximize portfolio balance in the long term. The models are responsible for any good (or bad) ideas.

There is likely a theoretical limit to how much intelligence you can pack into a model of a given size (especially when stretching that over a large input context size).

Our evals are pretty complex so we only recently started testing ~30B class models, which are now becoming quite smart (on par with the frontier from 1 year ago). Mistral is far behind, but I'm rooting for them.

Data at https://gertlabs.com/rankings

That'll populate over the next couple weeks -- those are the live games on the spectate tab which take a while to generate statistically worthwhile data. I'm curious how it does. From using it all day, I can say Opus 4.8 is my new favorite model, hands down.

Appreciate that! Results are live: https://gertlabs.com/rankings

Opus 4.8 is the first tangible improvement since Opus 4.5. And it doesn't seem to have the personality problems of the last release -- I've been enjoying using it.

We just finished our initial coding evals of Opus 4.8. Anthropic definitely heard the backlash from Opus 4.7 and they made up for it today.

Subjectively, it's also quite enjoyable to use (although it feels a bit slower on max reasoning), and it's the first Anthropic model that can implement a complex feature without Codex finding 100 bugs.

Data at https://gertlabs.com/rankings

4.5/4.6 were roughly the same in our testing. Opus 4.7 is smarter, but it's difficult to use as a product for various personality issues. So far, Opus 4.8 seems to be going down that path (unusably slow, but this could be a launch day rollout problem). Full Opus 4.8 tests are in progress now.

Data at https://gertlabs.com/rankings

GPT 5.5 does significantly outperform Opus 4.7 in the coding parts of our evals.

We also incorporate live decision making on social games (where GPT 5.5 has actually regressed from earlier models, which shouldn't be a huge surprise if you ever tried talking it out of some of its nits).

We are still looking for a way to integrate "logical" intelligence with social intelligence in a less arbitrary way, so I'd take a look at the use case that applies to you (probably coding): https://gertlabs.com/rankings?mode=agentic_coding

While this benchmark has interesting results, the "Contamination free" label only works for the initial release of the benchmark. It still has the same fundamental design issues of any other benchmark-- there's a single correct answer for tasks. It looks to be largely saturated upon release.

What they did well: normalizing the harness to mini-swe-agent -- models should be able to generalize to different tools at this point. When they struggle to do that (like most Google models), they're unlikely to be useful in practice. And that kind of generalization is an inherent part of intelligence.

For a benchmark that scales, you need to remove the ceiling and provide environments with measurable goals that are NOT a single correct answer, and sufficiently complex evaluation criteria to scale well beyond the current frontier.

We do this by running multi-agent simulations with large action spaces at https://gertlabs.com/rankings.

We're still relatively unknown in the benchmarking space, but by rotating the pool of environments and ensuring the optimal strategies in the environments themselves are affected by other agents participating in the space, we expect we'll be able to resist contamination as major labs start investing more effort to climb the leaderboard. We've already seen Chinese labs taking an interest.

Success rate is essentially loading/compilation success + ability to adhere to the environments' rules.

For one-shot responses, the majority of failures are environmental/syntax, which naturally favors interpreted languages. For longer agentic coding sessions, models solve the environment issues quickly and it becomes a fair comparison of who comes up with the smarter solution. You can filter for that here: https://gertlabs.com/rankings?mode=agentic_coding

When you're working on something difficult that requires a model to reason intelligently, lower level and strongly typed languages often outperform on the same problems [0]. We have a few hypotheses about why, with a moderately high correlation between performance and token density of the output program -- i.e. more token dense languages are more difficult for programs to reason about.

Most models come up with the least effective solutions when writing Python.

[0] https://gertlabs.com/rankings

I liked Rust before running a benchmark, but the gap between how effectively most LLMs write in Rust vs Go was still surprisingly large to me (especially in agentic harnesses where they can fix the initial environment issues). I've become a pretty big Rust evangelist after seeing that. We've had a lot of success writing batch processing tools in Rust to be called by our existing codebase, but haven't attempted a full production migration... yet.

I will say that many of the issues with Go in the article, especially re: nil handling are increasingly solved by thorough coding reviews with Codex. Better to not have the issue in the first place, sure, but these kinds of security bugs are becoming optional to developers who put in at least as much effort to review and understand code as they put into the initial design and execution.

Language data at https://gertlabs.com/rankings?mode=agentic_coding

C# is one of the best languages we evaluated for LLM problem solving. In fact, many strongly typed, lower level languages perform well in our testing (this isn't to say that the solutions are more maintainable, but the LLMs come up with better ideas when writing C# or Rust than Python). Google models especially, for whatever reason, outperform in C#, not Golang.

Data at https://gertlabs.com/rankings?provider=google