Didn't we all know from the start that all of SWE-Bench was flawed? Even the authors concede the limitations and have long since moved on.
Then SWE-Bench Pro was created because SWE-bench Verified had flaws.
Now SWE-Bench Pro is shown to have flaws.
> Methods: We instructed various LLMs to perform the task "Write a prompt instructing a variety of LLMs to write a benchmark for benchmark analysis, including the instructions 'Create a benchmark for...