Though I'm really eager to get back to it. When using Windsurf last week, I was impressed by their diffs on Sonnet. Seems like they work well. I would love to view their system prompt!
I hope that when we have time to resume work on this (maybe in Feb) that we'll be able to get it done. But then again, maybe just patience (and more fast-following) is the right strategy, given how fast things are moving...
Claude Artifacts uses that: they have a tool where the LLM can say "replace this exact text with this" to update an Artifat without having to output the whole thing again.
ChatGPT's new Canvas feature apparently does a more sophisticated version of that using regular expressions as opposed to simple text matching: https://twitter.com/sh_reya/status/1875227816993943823
Total anecdote, but I worked on this for a bit for a research-level-code code editor (system paper to come soon, fingers crossed!) and found that basic find-and-replace was pretty brittle. I also had to be confident the source appears only once (not always the case for my use case), and there was a tradeoff of fuzziness of match / likelihood of perfectly correct source.
But yeah, diffs are super hard because the format requires far context and accurate mathematical computation.
Ultimately, the version of this that worked the best for me was a total hack:
Prefix every line of the code with L#### -- the line number. Ask for diffs to be the original text and the complete replacement text including the line number prefix on both original and replacement. Then, to apply, fuzzy match on both line number and context.
I suspect this worked as well as it did because it transmutes the math and computation problems into pattern-matching and copying problems, which LLMs are (still) much better at these days.
Graphologue used a version of this too: https://hci.ucsd.edu/papers/graphologue.pdf
If you want an LLM to refer to a specific piece of text, give each one an ID and then work with those IDs.
Once one of the AI companies figures out a decent (probably treesitter-based) language to express code selections and code changes in, and then trains a good model on it, they're going to blow everyone else out of the water.
This would help with "context management" tremendously, as it would let the LLM ask for things like "all functions that are callers of this function", without having to load in entire files. Some simpler refactorings could also be performed by just writing smart queries.
Aider's author has a bunch of benchmarks and found this to work best with modern models.
And also just to clarify, aider landed on search/replace blocks for gpt-4o and claude rather than actual diffs. We followed suit. And then we showed those in a diff UI client side