Claude 3 beats GPT-4 on Aider's code editing benchmark
aider.chat
aider.chat
Claude 3 Opus outperforms all of OpenAI’s models, making it the best available model for pair programming with AI.
The new claude-3-opus-20240229 model got the highest score ever on this benchmark, completing 68.4% of the tasks. While Opus got the highest score, it was only a few points higher than the GPT-4 Turbo results. Given the extra costs of Opus and the slower response times, it remains to be seen which is the most practical model for daily coding use.
Only the weakest models like GPT-3.5 and Sonnet currently rely on the "whole" editing format. Most local LLMs struggle even with the whole format, although some do ok (but far worse than GPT-3.5/Sonnet).
The original gpt-4-0613 uses the search/replace "diff" format. The new Opus model works best with this format too.
The GPT-4 Turbo models work best with the unified diff "udiff" format. This format was created because it discourages GPT-4 Turbo from being "lazy" when coding.
Being able to handle diff or udiff is a big advantage -- they save a lot of tokens, money and time. I don't think it would be wise to use whole with any model than can handle one of the diff formats.
All that said, Opus' score climbs from 68.4->70.7% if you switch from diff->whole. I don't think that gain is worth the pragmatic downsides of the whole edit format. But you're welcome to run aider with `--edit-format whole` if you'd like!