• Terminal Bench: 56.9 → 82.7 (+25.8)
• Toolathlon: 51.8 → 70.3 (+18.5)
Compared to GPT-5.6 Terra:
• Terminal Bench: Flash 82.7 vs Terra 78.4
• Toolathlon: Flash 70.3 vs Terra 53.1
• DeepSWE: Flash 54.4 vs Terra 69.6
• Agents' Last Exam: Flash 25.2 vs Terra 50.4
Trading blows with Terra, which is pretty interesting. No clear winner on these benchmarks, and wildy differeing scores. Very interesting!
Terra 87.4
> 78.4
The real score is always the official benchmark.
We need to see later if DS4 flash 0731 is going to maintain the score but we need to look at the official benchmarks.
Already seen a PR for DeepSWE to update the benchmark with 0731, so we can verify claimed vs official.
They're literally comparing the previous version of the same model with the new one. It's based on the same architecture, same pre-trained model, just different post-training. It doesn't get more apples to apples than this.
The performance changes are so big with the right harness that is makes sense to engineer the harness and fine-tune the model to one another from the start.