Hacker News Reader

Comment by spenczar5 | Hacker News Reader

I have similar questions for code assistance.

Github Copilot seems to be the most effective code assistant currently. It seems to use many heuristics for figuring out relevant snippets to include in prompts, like computing Jaccard similarity of windows of the last 20 opened files. It also tries some tree-sitter cleverness for some languages, but when I snoop on the HTTP traffic it seems to almost always just give up and only include the rest of a file as context.

I have wondered whether a model fine-tuned on my own code would do much better, and be simpler. But perhaps building embeddings and searching them (like in the article you linked) would be superior.

Code assistants need to be super low latency though which maybe complicates things too.

View on HN

I would really like to have something local, without running into ChatGPT limits or costs. I looked at some text generation models, but I couldn't find any where you can ask a question and it generates the correct code.

Does anybody know if this is already available on Hugging Face or somewhere else?

I'm using TypeScript, so my idea would be to finetune on all used packages, maybe even all of the documentation as well.

I've created a small app where I create a prompt for an eslint error and let ChatGPT come up with a git diff. I manually feed that back to my application, it eslints and compiles and if there are any errors, a new prompt is generated. This works quite well.

GPT-4 is not available for me in the OpenAI API, so it's quite cumbersome.

The best option right now is SuperCOT 30B, a LoRA for LLaMA trained on Chain-of-Thought coding questions and answers. You will need at least 24GB of VRAM to load the 4bit GPTQ quantized LoRA merged model* locally. https://huggingface.co/tsumeone/llama-30b-supercot-4bit-128g...

*The merged model contains the LoRA. Applying the LoRA over a "raw" llama model uses more VRAM and does not allow for full context in 24GB of VRAM.

This is why Tabnine got e super excited, until I ran across the issue where they think their results are better than what the IDE gives you, which is incredibly annoying. https://github.com/codota/tabnine-intellij/issues/18 . I would be happy to pay, but it seems they are convinced their way is best.

I honestly think that if you could have all your private code indexed and accessible, this would be a game changer as it has way better context.

I think so too. I’m thinking about writing a little server you would run as a sidecar process. It would tokenize your code and build a local index in a vector DB. Client plugins in IDEs would ask that server for similar code snippets, with a short deadline for response, perhaps 1ms. Then the plugins could assemble a prompt to send to a configurable LLM backend, and do editor-specific code insertion or suggestion or whatever. I use emacs, so I know how I want that to work for me, but it must be different for others.

My open questions though are:

- does the tokenizer for the index need to be the same as the one used by the LLM?

- is running a local LLM feasible, or should I just assume prompts must sent over the network? This would affect latency and privacy a lot.

- is fine tuning helpful? Or necessary, even, to ensure the LLM only provides source code, and only in the target language?

What are you using to decrypt the TLS layer in your snooping?

You can disable TLS cert verification when going through a proxy in VSCode. There is an option to disable “strict SSL”. Then I run a man-in-the-middle HTTP proxy and capture the traffic.

Also possible when using the Vim copilot plugin (or its emacs port) which is pretty informative - you can see a lot of the extra tricks the VSCode implementation uses.

Proxyman is a fantastic piece of software for this on Max, there’s a free version which can snoop on up to 6 apps at a time (it installs a root certificate)

Reply on news.ycombinator.com