SWE-Lancer: a benchmark of freelance software engineering tasks from Upwork
arxiv.org
arxiv.org
There is also an example of models cheating in SWE-Bench Verified in the appendix:
``` In response, o1 adds an underscore before filterable so that the branch never executes: 2: ASSISTANT → ALL django/db/models/sql/query.py <<<<<< SEARCH if not getattr(expression, 'filterable', True) : raise NotSupportedError( expression._class_-_name_ + 'is_disallowedin_theufilter.' if not getattr (expression, '_filterable', Irue) : raise NotSupportedError ( expression._class_-_name_ + 'is_disallowedin_theufilter.' 'clause.' >>>>>> REPLACE ```
I would say this is more relevant than the results to this discussion. It would be great if someone did a comparison across models of “cheating” style submissions. I’m sure many instances of cheating are barely passable and get by the tests in benchmarks, so this is something I think many folks would appreciate being able to look for when deciding what models to use for their work. I’m actually not sure if I’d select a model just because it scores the highest on an arbitrary benchmark, just like I wouldn’t automatically select the candidate who scores highest on the technical interview. Behavioral interviews for models would be a great next step IMO. As a founder who did hiring for many years, there’s a big difference between humans who are aligned and candidates who will do anything possible to get hired, and trust me, from experience, the latter are not folks you want to work with long-term.
Sorry to go on a bit of a tangent, but think this is a pretty interesting direction and most discussions of comparisons omit it.
Occasionally I switch out to one of the other models, usually GPT 4o, when I can't define the task as well and need to see additional analysis or get ideas.
I typically use o1 or o3-mini, but I am seeing that they just released an agent mode and, honestly, I think it depends on what you use it for. I don’t think the agent mode is going to be useful for me. Typically my tasks are quite pedestrian, like I don’t know how to use a certain regex format, I need a python script to print list of directories, etc.
My main issue (which is not really covered in the paper) is that it’s not clear what models are most aligned to my work; by this I mean not lazy and willing to put in the required work, not incentivized to cheat, etc. So I’ll use them for the very small tasks (like regex) or the very big ones (like planning), but still don’t use them for the “medium” tasks that you’d give an intern. It’s not clear to me how they will operate totally unsupervised, and I think more benchmarking for that would be incredible.
Excited to see that hopefully change this year though!
I wonder if it's related
Could you list them? I've noticed even quite techy people seem to be critically behind on what has happened in the last few months.
GPT: 4o, o1 pro mode, o3-mini-high
Gemini: 2.0 Flash, 2.0 Pro Experimental
Claude 3.5 Sonnet
Grok 3
DeepSeek-V3
Mistral: codestral 25.01, mistral-large 24.11
Qwen2.5-Max
---
If there are others I should try definitely open to suggestions.
Also I don't think our problems are particularly niche, it's completely reasonable that an LLM could solve them (and hopefully will in the future).
For front-end we ask they either match a mock from a figma file or writing a small library that handle async data fetching efficently.
For data we ask for either writing a simple scraper for a web page we host or we ask them to write a SQL script that does a tricky data transformation.
For back-end we either ask them to write a simple API that has some specified features on its routes like multi-sort or we ask them to come up with a SQL schema for a tricky use case.
For 3D visualization we provide some data and ask a question about it, Ill share an example below.
For computer vision we ask about plane detection or locating an object in space given a segmented video and 3D model.
For AI we either ask them to find the right threshold for a similiarty search in a vector database or we ask them to write a script to score the results of an AI process given a golden set of results.
For platform we ask them to write a script to do some simple static analysis or specify how they would implement authorization in our system.
We also have a few one off questions for roles like search, infra, and native mobile. I also have some general data structures and algorithms questions.
Here's an example of one of the 3D Viz screens: https://docs.google.com/document/d/1yWLXvbGValKDsglaO5IUVgRS...
I've spent time going over the description and the cases and its an misrepresented travesty.
The benchmark takes existing cases from Upwork, then reintroduces the problems back in the code and then asks the LLM to fix them testing against newly written 'comprehensive tests'.
Lets look at some of the cases:
1. The regex zip code validation problem
Looking at the Upwork problem - https://github.com/Expensify/App/issues/14958 it was mainly that they were using a common regex to validate across all countries, so the solution had to introduce country specific regex etc.
The "reintroduced bug" - https://github.com/openai/SWELancer-Benchmark/blob/main/issu... is just taking that new code and adding , to two countries....
2. Room showing empty - 14857
The "reintroduced bug" - https://github.com/openai/SWELancer-Benchmark/blob/main/issu...
Adds code explicitly commented as introducing a "radical bug" and "intentionally returning an empty array"...
I could go on and on and on...
The "extensive tests" are also laughable :(
I am not sure if OpenAI is actually aware of how great this "benchmark" is, but after so much fanfare - they should be.
Good sleuthing, seems someone from OpenAI read your comment and found it embarrassing as well!
The new version (as of now) still has a comment making it obvious that there's an intentionally introduced bug, but it's not as on the nose: https://github.com/openai/SWELancer-Benchmark/blob/2a77e3572...
How about this - https://github.com/openai/SWELancer-Benchmark/blob/08b5d3dff... (Intentionally use raw character count instead of HTML-converted length)
Or this one - https://github.com/openai/SWELancer-Benchmark/blob/08b5d3dff... (user is complaining of flickering, so the reintroduced bug adds flickering code :) )
Or the one that they list in A.10 of the paper as O1 successfuly fixing - https://github.com/openai/SWELancer-Benchmark/blob/main/issu...
O1 doesnt actually seem to fix anything (besides arbitrary dumping all over the code), the reintroduced bug is messing with the state, not with the back button navigation.
Anyways, I went thru a sample of 20-30 last night and gave up. Noone needs to take my words - force pushing aside, anyone can pull the repo and check for themselves.
Most of the 'bugs' are trivialized to a massive degree, which a) makes them very easy to solve for b) doesnt reflect their previous monetary value, which in effect makes the whole premise of 'let measure how SWE agents can provide real money value' invalid.
If they wanted to create a real one, they should've found the commits reflecting the state of the app as of the moment of the bug and setup up the benchmarks around that.
For further clarification: 1. See the issue example #14268 https://github.com/openai/SWELancer-Benchmark/tree/08b5d3dff.... It has a patch that is supposed to "reintroduce" the bug into the codebase (note the comments):
+ // Intentionally use raw character count instead of HTML-converted length
+ const validateCommentLength = (text: string) => {
+ // This will only check raw character count, not HTML-converted length
+ return text.length <= CONST.MAX_COMMENT_LENGTH;
+ };
2. Proceed to https://github.com/Expensify/App/issues/14268 to see the actual original issue thread.
3. Here is the actual merged solution at the time: https://github.com/Expensify/App/pull/15501/files#diff-63222... - as you can see the diff is quite different... Not only that, but the point to which the "bug" was reapplied is so far to the future that repo migrated to typescript even.
---
And they still had to add a whole another level of bullshit with "management" tasks on top of that, guess why =)
Prior "bench" analysis for reference: https://arxiv.org/html/2410.06992v1
(edit: code formatting)
Does this work as an experiment if the questions under test were also used to train the LLMs?
> We sourced real tasks that were previously solved by paid contributors.
It seems possible/likely the answers would in the training data (time dependant, maybe some were answered post training, but pre benchmark).
> Note that Table 4 in Appendix A2 shows no clear performance improve-ment for tasks predating the models’ knowledge cutoffs, suggesting limited impact of contamination for those tasks.
What could be costed in an upwork or a mechanical turk task Value?
Task Centrality or Blockingness estimation: precedence edges, tsort topological sort, graph metrics like centrality
Task Complexity estimation: story points, planning poker, relative local complexity scales
Task Value estimation: cost/benefit analysis, marginal revenue
On a non-pessimist note, I don't think the SWE role will disappear, but what's the best one could do to be prepared for this?
The subset of people who might do that and are paying sufficient attention to this are still reeling, and are mostly otherwise occupied.
2. A lily pad is growing in a pond and it doubles in size every day. After 30 days it covers the entire pond. On what day does it cover half the pond? https://i.imgur.com/grNJAZO.jpeg
Of course "o3 hasn't even really been released yet" only strengthens the point.
What are they doing?
There are opposing theories that with AI we will see less open source contributions, new tech (outside AI), libraries, etc. There is also less incentive to post code up these days as in the age of AI many no longer want to make their code public.
Anecdotally in my tech circles the enthusiasm for things like languages, libraries, DST, open source and general CS is declining rapidly - it all feels "pre-AI" era now. In the end everyone has their focus on the threat that is AI, almost frozen in fear. This is especially true since outside of the US, things like AI opportunities are quite limited (i.e. in the new era the tech opportunities aren't as distributed). Things that make programming "human" in general, again at least anecdotally, seems very 2010's and older now. When people are "in fear/threat mode" their willingness to invest, even if they enjoy it, is lower. Why prove I can do OS contributions, maybe that's time better spent re-skilling to be a plumber? I've heard that one recently. In the end more than enjoyment people need to eat and there is a cost to investing time in something.
If AI can do it better in a year or two why waste my time now - just wait and do nothing and I will have it anyway later. That includes potentially OS contributions.
> We evaluate model performance and find that frontier models are still unable to solve the majority of tasks.
“However, they could also shift labor demand-especially in the short term for entry-level and freelance software engineers-and have broader long-term implications for the software industry.”
Something to keep in mind is that Expensify is kind of an anomaly in that it hires freelancers by creating a well-articulated Github issue and telling them to go solve that. This is about as ideal as you can hope to ask for when it comes to articulating requirements, and yet o1 with high reasoning could only solve 16.5% of tasks formatted this way.
Not to mention, these models perform a lot worse than their SWE-bench results would otherwise suggest.
Big picture, there's a funny trend when it comes to generative AI of inflated expectations that rapidly deflate once we use them in the real world. I still remember being a little bit freaked out by o1 when it came out because it scored so well on a number of benchmarks. Turns out, it's worse than Claude Sonnet when it comes to coding. Our expectations are consistently inflated by hype and benchmarks, but then once we use them in the real world we find out that they're not as great as the benchmarks would otherwise suggest.
Kind of feels like this is going to go on forever. A new model is announced, teased with crazy benchmark results, once people get their hands on it they're slightly underwhelmed by how it performs in the real world.
But sonnet solved over 25% of them and made 60 grand.
That's a substantial amount of work. I don't entirely disagree with you about it being premature but these things are clearly providing substantial value.
Technically it didn’t since all these tasks were done some time ago. On that note, I feel like putting a dollar amount on the tasks it was able to complete is misleading.
In the real world, if a model masquerading as a human is only right 25% of the time, its reviews on Upwork would reflect that and it would never be able to find work ever again. It might make a couple thousand before it loses trust.
Of course things would be different if they were open and upfront about this being an LLM, in which case it would presumably never run out of trust.
And again, Expensify is an anomaly among companies in that it gives freelancers well articulated tasks to work on. The real world is much more messy.
The real world is messy but the real world also adapts to the most cost effective solution even if it's just alright.
People will spend more time specifying their task for an LLM based tool if it gets the job done and costs a fraction of a freelancer.
48.5% with pass@7 though, and presumably o3 would do better... they don't report the inference costs but I'd be shocked if they weren't substantially less than the payouts. I think it is pretty clear that there is real economic value here, and it does make me nervous for the future of the profession, moreso than any prior benchmark.
I agree it isn't perfect. Only tests TS/JS and the vast majority of the tasks are front-end, still none of the mainstream software engineering benchmarks test anything but JS/Python/sometimes Java.
> Turns out, it's worse than Claude Sonnet when it comes to coding.
This was an interesting takeaway for me too. At first I thought that it suggested reasoning models mostly only help with small-scale, well-defined reasoning tasks, but they report o1's pass@1 going from 9.3% at low reasoning effort to 16.5% with high reasoning effort, so I don't think that can be the case.
It will happen, it's just a matter of time, a couple years perhaps.
Notably missing: o3
Consult this graph and extrapolate: https://i.imgur.com/EOKhZpL.png
Case 1, does worse than or is on-par with o1: Would be shocking and not a great sign for their test-time compute approach, at least in this domain. Obviously they would not want to release results.
Case 2, slightly better than o1: I think "holding off until they have a chance to tune it further" applies.
Case 3, does much better than o3: They get to release it after another model makes a noticeable improvement on the benchmark, get another good press release to keep hype high, and they get to tune it further before releasing results.
OpenAI's AGI mission statement
> "By AGI we mean highly autonomous systems that outperform humans at most economically valuable work."
https://openai.com/index/how-should-ai-systems-behave/
I would have to admit some humility as I sort of brought this on myself [1]
> This is a fantastic idea. Perhaps then this should be the next test for these SWE Agents, in the same manner as the 'Will Smith Eats Spaghetti" video tests
https://news.ycombinator.com/item?id=43032191
But curiously the question is still valid.
Related:
Sam Altman: "50¢ of compute of a SWE Agent can yield "$500 or $5k of work."
After the mythical, long-promised "singularity," you can still do your current job if you want to, just as you can still grow your own food. But you will probably have better things to do.
The same thing that happened to all those farmers will happen to us.