Hacker News Reader

Comment by LVB | Hacker News Reader

I'm not very plugged into how to use these models, but I do love and pay for both ChatGPT and GitHub Copilot. How does one take a model like this (or a smaller version) and leverage it in VS Code? There's a dizzying array of GPT wrapper extensions for VS Code, many of which either seem like kind of junk (10 d/ls, no updates in a year), or just lead to another paid plan, at which point I might as well just keep my GH Copilot. Curious what others are doing here for Copilot-esque code completion without Copilot.

View on HN

You can try it with Sourcegraph Cody. https://sourcegraph.com/cody

And instructions on how to change the provider to use Ollama w/ whatever model you want:

Install and run Ollama - Put ollama in your $PATH. E.g. ln -s ./ollama /usr/local/bin/ollama.

- Download Code Llama 70b: ollama pull codellama:70b

- Update Cody's VS Code settings to use the unstable-ollama autocomplete provider.

- Confirm Cody uses Ollama by looking at the Cody output channel or the autocomplete trace view (in the command palette).

- Update the cody settings to use "codellama:70b" as the ollama model

https://github.com/sourcegraph/cody/pull/2635

Hey, thanks for the tip.

One issue, though: I took a look at the Cody website and it looks like one can't have unlimited completions even when self-hosting a LLM.

I understand you guys have a business model and need to make money out of it. I'm just asking because I work as a teacher and I have students who can't pay an extra subscription and/or students who want to hack into stuff.

https://continue.dev/ is a good place to start.

Continue doesn’t support tab completion like Copilot yet.

A pull/merge request is being worked on: https://github.com/continuedev/continue/pull/758

Release coming later this week!

Thanks! I'm going to recommend it for my students who are interested in tinkering with LLMs.

Thank you for your efforts

Bonus points for being able to use local models!

This looks really good..

It's great. It's super easy to install ollama locally, `ollama run <preferred model>`, change the continue config to point to it, and it just works. It even has an offline option by disabling telemetry.

> Windows coming soon

ugh, not so easy.

Yes, that is certainly a downside I forgot to mention. Sorry to get your hopes up.

beat me to the punch : )

I’ve been working on continue.dev, which is completely free to use with your own Ollama instance / TogetherAI key, or for a while with ours.

Was testing with Codellama-70b this morning and it’s clearly a step up from other OS models

How do you test a 70B model locally? I've tried to query, but the response is super slow.

Personally I was testing with TogetherAI because I don't have the specs for a local 70b. Using quantized versions helps (Ollama's downloads 4-bit by default, you can get down to 2), but it would still require a higher-end Mac. Highly recommend Together, it runs quite quickly and is $0.9/million tokens

What’s the advantage of Together? The price is about the price of GPT 3.5 Turbo ($1/mil tokens is $0.001/thousand tokens), which has the advantage of wide ecosystem and support.

are there any docs on setting up togetherAI with continue.dev? would be interested in checking that out as an alternative to OpenAI for experimenting with larger models that won't run/run well on a m1 max.

Definitely, here is a brief reference page for the Together provider: https://continue.dev/docs/reference/Model%20Providers/togeth..., and a higher-level explanation of model configuration here: https://continue.dev/docs/model-setup/overview

Yeah, CPU inference is incredibly slow, especially as the context grows. 4-bit quantized on an A6000 should in theory work.

If those rent-seeking bastards at NVidia hadn't killed NVL on the 4090, you could do it on two linked 4090s for only $4k, but we have to live under the thumb of monopolists until such time as AMD 1. catches up on hardware and 2. fixes their software support.

There are some projects that let you run a self-hosted Copilot server, then you set a proxy for the official Copilot extension.

https://github.com/fauxpilot/fauxpilot

https://github.com/danielgross/localpilot

When I was setting up a local LLM to play with I stood up my own Open AI API compatible server using llama-cpp-python. I installed the Copilot extension and set OverrideProxyUrl in the advanced settings to point to my local server, but CoPilot obstinately refused to let me do anything until I’d signed in to GitHub to prove that I had a subscription.

I don’t _believe_ that either of these lets you bypass that restriction (although I’d love to be proven wrong), so if you don’t want to sign up for a subscription you’ll need to use something like Continue.

This plugin just installs in Jetbrains IDE and sets up a local llama.cpp server https://plugins.jetbrains.com/plugin/21056-codegpt

I tried fauxpilot to make it work on my own llama.cpp instance, but didn't work out of the box. Filed a github issue, but did not get any traction. Eventually gave up on it. This was around 5 months ago. Things might have improved by now.

Free Bard is better than free ChatGPT... Not sure about paid versions

Bard censorship is annoying. One thing I've found (free) Bard to better than the rest at is summarizing book chapters, manuals, and docs. It is also surprisingly good at translation (X to English), as it often adds context to what its translating.

With careful prompt engineering, you can get a lot out of free Bard except when its censored.

Not sure why you talk about Bard's censorship and not Chatgpt's but cause in my experiences it is much worst.

I'm learning Rust. It seems to me that Bard is better than OpenAI 4 I use for free at work.

I use the plugin Twinny in conjunction with ollama to host the models. Easy setup and quite powerful assistance. You need a decent rig though, since you don't want any latency for features like autocomplete.

But even if you don't have a faster rig, you can still leverage it for slower tasks to generate docs or tests.

Twinny should really be more popular, didn't find a more powerful no-bullshit plugin for VSCode.

You can download it and run it with [this](https://github.com/oobabooga/text-generation-webui). There's an API mode that you could leverage from your VS Code extension.

I've been using Cody by Sourcegraph and liking it so far.

https://sourcegraph.com/cody

Reply on news.ycombinator.com