6,873 karma · joined August 20, 2020
contact: hn@kortlepel.com
> On FrontierCode, which evaluates whether coding agents produce changes ready to merge into real codebases, GPT‑6 Sol improves substantially over GPT‑5.6 Sol, and is able to match Claude Fable 5.1 xhigh at much lower cost.
I continue to appreciate OpenAI's attempt at some honesty here, showing that they are capable enough and have skilled engineers to a point where they can recognize that slop is hated for good reason, and that there is a real issue. Compare this to anthropic, where e.g. in the Opus 5.5 announcement[1] one of the first points on the page is
> One tester completed a 680,000-line code migration in less than a day—work that would have taken an engineering team weeks. It’s good at finding and fixing inefficiencies in software: when we asked it to cut load times across every page of a web app, Opus 5.5 succeeded 39 of 40 times, while Opus 5 made smaller improvements that also altered the app’s behavior. A different tester had several Claude models build a game from a single prompt; Opus 5.5 scored higher than any other model on the strength of its graphics and polish.
This is the kind of shit that is the very reason why I stick to OpenAI and deepseek. OpenAI is simply more honest and reasonable about their models' capabilities, while delivering models that still have solid value.
Notice how the OpenAI announcement doesn't make use of anecdotes.
When humans do this, they inadvertently learn something, too, but when an LLM reproduces or derives and implementation of a chess engine, it in no way implies that the LLM can follow the rules in its own "train of thought" and consistently apply the rules in its "head".
Let's say you want to evaluate my algebra skills. You make me solve some algebra challenges. If I then whip out a computer and write a calculator, or take some sticks and stones and take a couple hours to build an abacus, and then solve the algebraic challenges, this would not constitute a good solution, and would defeat the entire point of the test. If, instead, I do the algebra in my head or on paper, it might seem like there's no difference, but you can derive all sorts of information from that.
For example, you could time it, check for recurring errors I make, for interesting mistakes like mistaking 7 and 1 for one another due to bad hand-writing, etc.
If that was the goal, then me writing a calculator or crafting an abacus defeats the point of the test. Yes, me writing a calculator shows that I'm intelligent, and I understand the algebraic rules, but if the test is about applying the rules, I have not passed.
In the very same way, an LLM writing a chess engine to solve a chess benchmark that is all about LLM's reasoning capability is complete bogus and defeats the entire point.
Writing a well understood engine for a super popular problem does not count as reasoning about the problem.
If you let the LLM write a chess program, which it can ONLY do because there are already so many chess programs out there, then the benchmark becomes about recall of popular program source code, not chess.
Ads don't just steal your time either, they invade your head. You are influenced by them and you have no choice as long as you watch them.
And they look nice, feel nice. Some of them even have a glass/crystal back, so you can watch the mechanism, relentlessly ticking on.
Get an analog automatic watch. It doesn't need to be second-accurate. Everyone else is keeping track of nanoseconds, you'll be okay not knowing.
Its especially funny in parking here lots in Europe, where they barely fit. If you drive such an SUV, I have a suggestion: Let the computer do the parking, you can't park and you can't drive well either. Just let the clanker figure it out, it'll save everyone time.
In Pi (pi.dev), it tells me it's definitely Claude by Anthropic, via the API via curl it tells me it's "probably ChatGPT", its very funny.
Snark aside, the researchers working on this, who built the foundation, should get credit, and they are.
If an LLM helps you do that, then use it.
Have you seen LLMs build large, stable, pieces of software, that work well, with your own eyes? I have not. I have seen LLMs build small, well-understood things, and I've seen LLMs do super well with extremely well specified problems, but all the rest seems to be marketing (i.e. I couldn't observe or reproduce it).
Edit: I'm not talking about throwaway slop that just needs to run long enough to get funding.