Engineering over AI

Very true!

I do RAG for other types of structured data and this is fundamental to get relevant objects in your context.

My approach for code would be to create a graph structure with relationships between the different codepaths and expose a retrieval api through tools/function calling so that the LLM can query the codebase structure on top of doing semantic embedding similarity search and text similarity search.

You could also add a graph search for related elements for each element returned by the other search pipelines to increase the chance of having all the pieces of the puzzle in the context before using the LLM to solve the problem.

The other crucial thing to do would be to inspect dependencies (and their types, when possible) and maybe download documentation to offer tips that are accurate and not hallucinated.

Nowadays I get hallucinations for code generation as soon as things get hard, making LLM coding useful only for trivial code writing.

Analysing the code structure and dependencies would require plenty of work for each specific language, so it won't be a easy win like "just throwing RAG" - which is what the current players are doing to raise money - with mediocre results.

I feel like I just read the introduction to an interesting blog post.

I feel like this is both very right but also the million dollar question?

I don't think others necessarily quote-unquote "lost focus" on this problem, but it's not exactly easy to solve correctly, so in the meantime it's easier to create something with the next best approximation.

I think this is the idea behind sourcegraph's cody, trying to take their expertise in understanding codebases and ASTs and using it to guide the llm

Where's the rest of the article?

Typo of "codebase" as codebae: This is also the reason why the higher context window doesn’t matter. Even if you could feed your whole codebase into an LLM, you’d still face the same problem of missing the structural relationships of the codebae.

You would need to have smaller agents negotiate on behalf of their functional units.