Enhancing Code Completion for Rust in Cody | Hacker News Reader

Enhancing Code Completion for Rust in Cody

sourcegraph.com

82 pointsimnot40426 comments

Awesome to see our Rust fine-tune here on HN. We (Sourcegraph team) are here if anyone has questions or ideas for us!

BTW, Cody is open source (Apache 2): https://github.com/sourcegraph/cody.

Your Cody page doesn't answer a very obvious question: does the LLM run locally or is this going to send all my code to Sourcegraph? I assume that is a deliberate omission and the answer is the latter.

The default LLM is Claude 3 Sonnet hosted by Anthropic. However you can run local models via Ollama: https://sourcegraph.com/docs/cody/clients/install-vscode#sup...

Thank you, cody.dev kept giving me a white blank page.

Sorry about that! I just looked into this (also on the Sourcegraph team). If you get a chance, could you check if it's working now?

Just wanted to thank you, is a great product. I find it to have a better context and awareness of my code compared to copilot

From the article:

> Although most LLMs are trained on corpora that include several programming languages, we often observe differential performance across languages, especially languages like Rust that are not well represented in popular training datasets.

Interesting! My experience with Rust and Copilot is that Rust is actually a relatively strong language for Copilot, in many ways. I suspect that this might be a mix of different things:

- Is the training set code for Rust relatively high-quality?

- Do strongly-typed languages have a better chance of catching incorrect completions quickly?

My very limited experience with GPT and Rust is that the code it generates rarely satisfies the borrow checker out of the box. I’ve had much better luck with garbage collected languages.

Interesting. I find that Copilot is quite good with the borrow checker, at least if it can "see" the signature of the function that it's calling. Sometimes it inserts an extra "&" where it really shouldn't. That's a no-op, clippy catches it, and I take it out.

But I tend to write my code in a style that easily passes the borrow checker 99% of the time. I keep mutation very local, and rely on a more "functional" architecture. So Copilot is naturally going to write new code in the same borrow-checker-friendly style, which may keep it out of trouble.

When using ChatGPT, I've actually had more luck asking it to write Python, and then to translate the Python into Rust. ChatGPT seems to be better at algorithms in Python, and it's surprisingly good (in my personal experience) at inserting the fine details of Rust types and references when translating.

Does your more functional style involve making a lot of copies?

Cody is decent. It's my copilot of choice at the moment. It indexes your code base so it can give contextual responses.

What would make it better than decent for you?

1. Write the commit messages too. 2. Allow the user to refactor across selected portions of the entire project, not just one file.

For 1, set `"cody.experimental.commitMessage": true` in VS Code and then click the Cody icon above the commit message box in VS Code's source control sidebar. Let me know how that works for you (here or at https://community.sourcegraph.com).

For 2, we're working on it! And eventually we'll support cross-repository refactors in Cody using our batch changes (https://sourcegraph.com/docs/batch-changes). What kind of refactors do you want to make?

1. I use Jetbrains. I don't suppose experimental flags are supported there too?

2. A reasonable near-term goal could be to make changes in one file propagate across the code base, so it remains consistent. This requires analyzing the call graph, which is something Sourcegraph already knows how to do. The next step could be to execute multi-file refactors without specifying which files to modify; e.g., "add support for WebAuthn using our auth provider's SDK".

1. Ah, this commit message feature isn't available in JetBrains yet, unfortunately. Most things carry over since they both use the same underlying code, but this is more editor-specific.

2. Your "reasonable near-term goal" is right in line with what we are thinking. Basically, "auto-edits" is f(edit that meets deterministic trigger criteria) --> multi-file diff. Here are some examples we have in mind; any others you had in mind?

update a type signature --> ripple that change across to all call sites/value literals (kinda like symbolic rename does today, but for more than just the name)

rename something --> update the docs

change code with a comment nearby --> update the comment if it's now invalid

update a func impl --> propose a new test case

I’ve been using Cody and liking it, but one thing I noticed is there is no loop that even takes the data from the IDE and incorporates the IDE’s feedback into the code it generates. It’ll happily generate code the IDE draws red wiggle lines under and just continue on with its day. Would be nice to see a tighter integration

Yeah, that is needed to make everything more accurate. We are working on 3 related things to make Cody better understand the structure of your code and take feedback from the compiler (and other tools).

All are experimental and not enabled by default, but you can peek at them if you're interested.

1. (experimental) LSP-based context — https://github.com/sourcegraph/cody/pull/3493

2. (experimental) TypeScript tsc context — https://github.com/sourcegraph/cody/pull/3843

3. (experimental) OpenCtx, which can pull in diagnostics from tools outside your editor — https://openctx.org/

And then of course the agent to actually iterate on the generated code based on this feedback.

We're really excited about this stuff, and it'll arrive in Cody when we can get it to a high quality bar and prove it helps.

Awesome! I’ll take a look

arguably you can reduce even more latency by keeping the model on-device as well, but that would mean revealing the weights of the fine-tuned model.

If the user preferred reduced latency and had the RAM, is that an option?

the model is probably most of the "secret sauce" of cody, so if they gave that away people could just copy it around like mp3s. my guess

Completely incorrect, as Sourcegraph has not historically trained models and Cody swaps between many open source and 3rd party models.

Looking at their GitHub page or seems like they are using existing LLM services. It should be possible to modify cody to make it work with a local llm

You don’t have to modify anything. We support local LLMs for chat and completions with Ollama.

https://sourcegraph.com/blog/local-code-completion-with-olla...

This is true, but only if you have a GPU (/accelerator) comparable in performance to the one backing the service, or at least comparable after accounting for the local benefit. This is an expensive proposition because it will be sitting idle between completions and when you're not coding.

Not for this fine-tuned model yet, but Cody supports local models: https://sourcegraph.com/docs/cody/clients/install-vscode#sup....

I just used Cody with Ollama for local inference on a flight where the wifi was broken, and it never fails to blow my mind: https://x.com/sqs/status/1803269013310759236.

Reply on news.ycombinator.com