Staff SWE Bench: LLM doubts whether we should do any of this, calls the entire project into question, refuses to merge code, but is happy to delete it.
I suspect LLMs can do this just fine and probably better than us, but they do need to be trained specifically for it and I have a hard time coming up with good sources of training data for it.