Hacker News Reader

Comment by _boffin_ | Hacker News Reader

I will only want to use this if i'm able to host the LLM locally.

Currently using CodeGPT and running CodeBooga 34b @ 4bit on my work laptop. This is good enough for me, but i'd love some more advanced functionality, such as code completion.

CodeGPT

- Repo: https://github.com/carlrobertoh/CodeGPT

- Plugin: https://plugins.jetbrains.com/plugin/21056-codegpt

View on HN

Been using CodeGPT a while myself.

Great tool, send e-mail to the author, seems like a good person.

I actually hook it up to my Windows gaming rig. The 4090 gets the job done, when I'm not gaming. (I just start the server when I go work, and then stop it when I want to game. It isn't that big a deal. I haven't hit the use-case where I want to use the AI to develop game mods yet. But I'll cross that bridge when I get there. ;) )

Which model are you using? I tried the deepseek 34b or whatever, but it goes nuts at times. Also, what params do you have for Prompt Context Size and Max Completion Tokens?

Currently Phind_Phind-CodeLlama-34B-v2, gonna try CodeBooga because, I can... that's the point. Use what ya like.

4k/32k, if I remember right. An interesting thing to do is actually try to use a true coding model in a chat. I had one model puking out its stack overflow training data, down to including user IDs.

Oops.

Isn't that going to be extremely slow? I can only realistically run 7B 5-bit models on my RTX 3060, anything more and it offloads to the CPU. My responses go from almost-instantaneous to 3mins+.

It seems like it's running at comparable speeds to GPT4 prior to Turbo. I could be wrong, but what I'm trying to say, it ain't bad at all.

This is where the Mac world shines.

would a 32gb M2 Max be able to run a 34b-model?

> running CodeBooga 34b @ 4bit on my work laptop.

How much RAM or VRAM does that require?

It’ll be around 17GB + context. For longer contexts you’ll add a couple GBs.

Max ram is saying 22.72GB. They gave me a laptop with a M2 MAX and 64GB of ram. I'm loving the fact that i can toss any piece of code in there without having to worry about data privacy issues and or any retention policies of service providers.

In my experience the vastly oversimplified-beyond-actual-usability calculation is that you take the parameter number (34) and count that as the amount in GB you'd need for an 8 bit model. with a 4-bit quantization that's halved, so your ball park is 17 GB plus a few GB for context. You can run these in RAM, but it's not very fast with (if I had to guess) slightly less than 1 Token/sec. on most non-apple consumer devices.

Give it another year. In theory it's not going to be difficult, but sampling speed/power consumption is prohibitive for normal hardware at the moment.

Honestly, with a 3090 costing like $700 used. It is within reach of many on this site quite easily, assuming they have a machine to plug the 3090 into.

People seem to think the hardware for this stuff is way out there... it is getting to be within reach VERY quickly.

Yes, but if you're looking for code completion at the quality level of Codex/Github Copilot, that's still a bit off.

Can you show me the metrics for that, please.

Additionally, what's your company's IP policy? Do they let you send their code to a remote company or do they see that as a breach of security?

I'm sure there are all sort of companies with very strict isolation requirements. But your typical enterprise will run your code editor in cloud VDI, host on atlassian, deploy using atlassian, deploy on some other cloud provider and be developed by double outsourced junior developer who's got warezd windows licence. I fail to see how adding another party who says they explicitly don't collect your data will make any difference.

Don't fall into the trap of: It has to be the best or it is useless.

The question for me is: Does it help me get my job done. Does it assist me in ways I find useful.

I have been using github copilot for a while now, but when I first tested these, together with the code AI offering from Amazon, they were... kinda useless. Only after a few updates did it get to a point that was a net positive in my daily work. And I still occasionally check the local models, and they're still not on a level in their auto-complete capabilities that I'd like to use, at least last time I checked. But I work with other LLMs and the recent advances in speed and the stricter nature of code means that soon-ish we should have a good and fast local model, we're just not there yet.

I don't use them for auto-complete that may be the difference. I use them for various other things. Tests, etc. I don't find them 100% accurate. But it beats writing it from scratch.

Okay so maybe 5% of developers who use desktop will be willing to fork out on a GPU and go thru process of setting up something that costs $10 otherwise...

> sampling speed/power consumption is prohibitive for normal hardware at the moment.

For the top-end of existing consumer hardware for the purpose, that might be in the process of changing quite quickly:

https://huggingface.co/blog/optimum-nvidia

Reply on news.ycombinator.com