Hacker News Reader

Comment by doctoboggan | Hacker News Reader

I am always excited to see these Open Weight models released, I think its very good for the ecosystem and definitely has its place in many situations.

However since I use LLMs as a coding assistant (mostly via "rubber duck" debugging and new library exploration) I really don't want to use anything other than the absolutely best in class available now. That continues to be GPT4-turbo (or maybe Claude 3).

Does anyone know if there is any model out there that can be run locally and compete with GPT4-turbo? Or am I asking for something that is impossible?

View on HN

You're asking for something that doesn't exist yet. Command R+, WizardLM-2-8x22B probably come closest.

Do you mind my asking, if you're working on private codebases, how you go about using GPT/Claude as a code assistant? I'm just removing IP and pasting into their website's chat interface. I feel like there's got to be something better out there but I don't really know anyone else that's using AI code assistance at all.

Use the API (or a product that uses the API)

If you can trust Azure or AWS or GCP with your IP, you can get Claude 3 and GPT-4 Turbo through at least one of them

If your IP is so secret you can't do that, then I wouldn't imagine you'd be using the chat interface

I haven’t used it but I’ve heard https://cursor.sh/ might work?

Personally I don't paste anything. I ask for code examples that demonstrate what I want, and then I adapt it to my needs. It's definitely less powerful than directly sharing code, but it is what it is.

I also run a personal language model server, but that is far less capable than the models available as services. It can still be better than nothing for code O can't share with APIs.

I also use gpt.el a but for editor integration, but I honestly haven't workeded that into my workflow very much yet.

Unless you have the privilege of being an enterprise customer with an SLA guaranteeing privacy, there's not much you can do other than using local models. I believe OpenAI says they don't train based on API requests but that's more of a "trust me bro" than any kind of guarantee.

Team and Enterprise come with the non-training guarantee, free and premium do not. Pretty much anyone can sign up for Team (I have, and I'm not a company) but you need to buy at least 2 seats for a total of $50/m. The rate limits are much better with that as well though.

Vscode with GitHub copilot is great, been using it for about a year and a half, no complaints. The business tier allegedly doesn’t save/train on your data

I'm building Plandex (https://github.com/plandex-ai/plandex), a terminal-based AI coding tool which currently uses the OpenAI api--I'm working on support for Anthropic and OSS models right now and hoping I can ship it later today.

You can self-host it so that data is only going to the model provider (i.e. OpenAI) and nowhere else, and it gives you fine-grained control of context, so you can pick and choose exactly which files you want to load in. It's not going to pull in anything in the background that you don't want uploaded.

There's a contributor working on integration with local models and making some progress, so that will likely be an option in the future as well, but for now it should at least be a pretty big improvement for you compared to the copy-paste heavy ChatGPT workflow.

Very cool! I'll take a look.

I built a desktop tool to help reduce the amount of copy-pasting and improve the output quality for coding using ChatGPT or Claude: https://prompt.16x.engineer/

Nope, I don't even see what the excitement is for.

We seem to be in denial of the scaling problems we face in that we can't even beat out the 1 year model.

I subscribed and unsubscribed to Claude 3 in about an hour. It is just not better than chatGPT4.

It is incredible to me that with all the motivation and resources of Meta, the best they can do is to produce a language model that isn't worth the time to even bother trying if a chatGPT4 subscriber.

Reply on news.ycombinator.com