Just to nitpick the math. If you are going to fire 50% of the company, the AI tools should actually make the remaining people 100% more efficient, not 50% :)
167 karma · joined February 19, 2025
swe bench pro public is newer, but its not live, so it will get slowly memorized as well. the private dataset is more interesting, as are the results there:
"We've deployed trillions of tokens across these agents toward a single goal. The system isn't perfectly efficient, but it's far more effective than we expected."
https://github.com/wilsonzlin/fastrender/blob/3e5bc78b075645...
At this point, its 1.5mlocs without the vendored crates (so basically excluding the js engine etc). If you compare that to Servo/Ladybird which are 300k locs each and actually happen to work, agents do love slinging slop.
The "in progress" build has a slightly different rendering but the same result
I couldn’t make it render the apple page that was on the Cursor promo. Maybe they’ve used some other build.
Looking at OAI API pricing, 5.2 Codex is $14 per 1 million output tokens. Which makes cool $14m for 1 trillion tokens (multiplied by whatever the plural is). For something that "kind of works".
Its a nice ad for OAI and Anysphere, but maybe next time - just donate the money to a browser team?
It a broken mess that probably implements 0.00001% of Excel. And its 1.2m locs.
With codebases developed in this way - either they need to figure out how agents are going to maintain them (in which case SWE as we know is dead - it will only be limited to those that can spend trillions of tokens, or they are going to remain weird demos.
I guess probably at some point, something compiled, but cba to try to find that commit. I guess they should've left it in a better state before doing that blog post.
The problem (which should be obvious) is that with a/b real you cant construct an exhaustive input/output set. The test case can just prove the presence of a bug, but not its absence.
Another category of problems that you cant just test and have to prove is concurrency problems.
And so forth and so on.
The models are gotten very good, but I rather have an obviously broken pile of crap that I can spot immediately, than something that is deep fried with RL to always succeed, but has subtle problems that someone will lgtm :( I guess its not much different with human written code, but the models seem to have weirdly inhuman failures - like, you would just skim some code, cause you just cant believe that anyone can do it wrong, and it turns out to be.
And yeah - if I had a crystal ball, I would be on my private island instead of hanging on HN :)
And also - doesn’t that make Zed (and other editors) pointless?
(note that if you look at individual slices, Opus is getting often outperformed by Sonnet).
Simon - you are an outlier in the sense that basically your job is to play with LLMs. You don't have stakeholders with requirements that they themselves don't understand, you don't have to go to meetings, deal with a team, shout at people, do PRs etc., etc. The whole SDLC/process of SWE is compressed for you.
from the old parser ast -> to json -> to new ast representation (that is basically again copy of the old one) -> to some new incomplete bytecode generation
im sure there is some good explanation, but....why?! :)
LCB Pro are leet code style questions and SWE bench verified is heavily benchmaxxed very old python tasks.
Its a very impressive model, but I think we have different views on what is complex.