What is considered SOTA for SWE benchmarks now?
Also I wonder if models playing dumb to prevent learning on outputs affected the score.
Also interesting that Claude edits files by writing and running Python scripts, is that efficient?
I doubt that, in these tasks, model restrictions to prevent training are affecting the results, not least because for both evals, the labs provided pre-release model access and have an incentive to be seen as favorably. In any case, I have not seen regressions to prevent distillations myself even when working on microscopic model training projects with LLM assistance, what I have however reliably and consistently seen is that some providers do train on popular evals and can underperform with minor changes to the task due to that.
Yes, harnesses, including Claude Code can prompt the models to write throwaway code to execute certain tasks, mostly Python, bash scripts or TS/JS, with there being some biases towards one over the other depending on the lab or specific model. Mainly for repetitive tool calls with no pre-existing/provided tools enabling it. Is in most instances a lot more efficient then a model e.g. doing a refactor that requires consistent variable renaming directly and around Opus 4.1/GPT-5, models have been trained to very consistently and accurately gauge when a task can benefit from such scratchpad scripts vs when that is inefficient/not useful.
Unless you want to tack on bpe enconding table to every llm context its pointless