kevinlu1248

I feel the same, benchmarking in general is a pain but a good benchmark for us could go a long way.

Like stochastically generate into an intermediate representation that can be procedurally compiled to code right?

This is really interesting, will take a look. So is this basically English to code translation?

Not a great answer but we are open-source so forking us is an option.

From our experience, single or few line replacements generally fine, since many of the changes are many few-line changes in multiple spots accross multiple files. We also provide surrounding context for the search-and-replace pairs, which helps with the model. Beyond 10 lines and the model also usually add the function headers in there which helps with the code generation.

I'm also curious, how are you guys evaluating the performance of your models?

Yup, saw a few papers about this over the past two years, using graph neural networks for code generation. There's also another thread below on this topic.

Edit: Here's some of the papers: https://arxiv.org/abs/1911.09983 and https://aclanthology.org/2021.findings-acl.384.pdf

Definitely. I prefer the sentence-transformers ones since they have been fine-tuned on codesearchnet. I'm also really excited about the latest gte models by Alibaba, their smallest model is the size of MiniLM L6 but beats MPNet.

We have a similar method under the hood, except it's purely text-based search-and-replace. The model decides what to replace. It seems to be consistent and is easy to implement.

Shot an email. Thanks!

This is interesting, I'm wondering what you mean by AST substitution. Is this like an agent that traverses the tree and picks what to edit? Is this language model based? Also, thankfully we don't support too many uncommon languages. The most recent ones we added support for are embedded templates (ERB/EJS for flask and Ruby) and mustache. Fortunately many uncommon languages are subsets of other languages.

We used to use line numbers but it became problematic so we switched over to search-and-replace pairs, which works significantly better. The only potential problems are with setting up a fuzzy search system since sometimes the search doesn't match exactly with the code (missing comments, etc.). We're going to write about our core algo and diff managing system soon.

Which app? We did notice a few visual errors a while back.

That would be awesome. Let me know when it's out and I'd love to try it out.

I'm glad to hear! Let me know if you need more questions or are interested in implementing this chunking algo.

Hey thanks for showing this dashboard. There's some crazy analytics in here, the tool looks awesome!

We got rid of it. Our new edit framework works around search-and-replace pairs with an example at https://github.com/sweepai/sweep/blob/d37dda3a626f09dea3b322...

From my experience, GPT-4 never makes syntax errors directly but when making edits to existing code it's harder to prevent these syntax errors from appearing. We used to add a second pass to check for these syntax errors.

It also frequently makes undefined variables and the like, however.

This is interesting. I'm taking a read on this.

That's interesting, I've seen a few papers about this. I'm personally curious about editing syntax trees using language models, since it would prevent syntax errors altogether.

Thanks! Would love to see this algorithm in LlamaIndex.

At Sweep, we don't want to replace developers, we want to augment them. Instead, developers will become more productive and can work on the exciting problems that matter. http://sweep.dev/

We gave our AI juinor dev access to Github Actions to run linters and tests, so it can fix syntax errors and avoid undefined variables.

Check out our work at http://sweep.dev/