The writing is very clearly on the wall.
On a non-pessimist note, I don't think the SWE role will disappear, but what's the best one could do to be prepared for this?
On a non-pessimist note, I don't think the SWE role will disappear, but what's the best one could do to be prepared for this?
Something to keep in mind is that Expensify is kind of an anomaly in that it hires freelancers by creating a well-articulated Github issue and telling them to go solve that. This is about as ideal as you can hope to ask for when it comes to articulating requirements, and yet o1 with high reasoning could only solve 16.5% of tasks formatted this way.
Not to mention, these models perform a lot worse than their SWE-bench results would otherwise suggest.
Big picture, there's a funny trend when it comes to generative AI of inflated expectations that rapidly deflate once we use them in the real world. I still remember being a little bit freaked out by o1 when it came out because it scored so well on a number of benchmarks. Turns out, it's worse than Claude Sonnet when it comes to coding. Our expectations are consistently inflated by hype and benchmarks, but then once we use them in the real world we find out that they're not as great as the benchmarks would otherwise suggest.
Kind of feels like this is going to go on forever. A new model is announced, teased with crazy benchmark results, once people get their hands on it they're slightly underwhelmed by how it performs in the real world.
But sonnet solved over 25% of them and made 60 grand.
That's a substantial amount of work. I don't entirely disagree with you about it being premature but these things are clearly providing substantial value.
Technically it didn’t since all these tasks were done some time ago. On that note, I feel like putting a dollar amount on the tasks it was able to complete is misleading.
In the real world, if a model masquerading as a human is only right 25% of the time, its reviews on Upwork would reflect that and it would never be able to find work ever again. It might make a couple thousand before it loses trust.
Of course things would be different if they were open and upfront about this being an LLM, in which case it would presumably never run out of trust.
And again, Expensify is an anomaly among companies in that it gives freelancers well articulated tasks to work on. The real world is much more messy.
The real world is messy but the real world also adapts to the most cost effective solution even if it's just alright.
People will spend more time specifying their task for an LLM based tool if it gets the job done and costs a fraction of a freelancer.
48.5% with pass@7 though, and presumably o3 would do better... they don't report the inference costs but I'd be shocked if they weren't substantially less than the payouts. I think it is pretty clear that there is real economic value here, and it does make me nervous for the future of the profession, moreso than any prior benchmark.
I agree it isn't perfect. Only tests TS/JS and the vast majority of the tasks are front-end, still none of the mainstream software engineering benchmarks test anything but JS/Python/sometimes Java.
> Turns out, it's worse than Claude Sonnet when it comes to coding.
This was an interesting takeaway for me too. At first I thought that it suggested reasoning models mostly only help with small-scale, well-defined reasoning tasks, but they report o1's pass@1 going from 9.3% at low reasoning effort to 16.5% with high reasoning effort, so I don't think that can be the case.
It will happen, it's just a matter of time, a couple years perhaps.
> We evaluate model performance and find that frontier models are still unable to solve the majority of tasks.
The subset of people who might do that and are paying sufficient attention to this are still reeling, and are mostly otherwise occupied.
2. A lily pad is growing in a pond and it doubles in size every day. After 30 days it covers the entire pond. On what day does it cover half the pond? https://i.imgur.com/grNJAZO.jpeg
Of course "o3 hasn't even really been released yet" only strengthens the point.
What are they doing?
There are opposing theories that with AI we will see less open source contributions, new tech (outside AI), libraries, etc. There is also less incentive to post code up these days as in the age of AI many no longer want to make their code public.
Anecdotally in my tech circles the enthusiasm for things like languages, libraries, DST, open source and general CS is declining rapidly - it all feels "pre-AI" era now. In the end everyone has their focus on the threat that is AI, almost frozen in fear. This is especially true since outside of the US, things like AI opportunities are quite limited (i.e. in the new era the tech opportunities aren't as distributed). Things that make programming "human" in general, again at least anecdotally, seems very 2010's and older now. When people are "in fear/threat mode" their willingness to invest, even if they enjoy it, is lower. Why prove I can do OS contributions, maybe that's time better spent re-skilling to be a plumber? I've heard that one recently. In the end more than enjoyment people need to eat and there is a cost to investing time in something.
If AI can do it better in a year or two why waste my time now - just wait and do nothing and I will have it anyway later. That includes potentially OS contributions.
“However, they could also shift labor demand-especially in the short term for entry-level and freelance software engineers-and have broader long-term implications for the software industry.”