Hacker News Reader

Comment by codybontecou | Hacker News Reader

I've been using [continue](https://continue.dev/) alongside Ollama. My go-to llm has been [deepseek-coder 7b](https://ollama.com/library/deepseek-coder). The setup feels as good as ChatGPT 4, local first, and overall, I enjoy it.

View on HN

I've tried code-llama with Ollama, along with Continue.dev and found it to be pretty good. The only downside is that I couldn't "productively" run the 70B version, even on my MBP with M3 Max with 36GB of RAM (which interestingly should be enough to hold quantized model weights). It was simply painfully slow. 34B one works good enough for most of my use-cases, so I am happy.

I tried to use codellama 34B and I think it is pretty bad. For Example I asked it to convert a comment into a docstring and it would hallucinate a whole function around it.

Same, CodeLlama 70B is known to suck. Deepseek is the best for coding so far in my experience, Mixtral 8x7B is another great contender (to be frank, for most tasks). Miqu is making a buzz, but so far I haven't tested it personally yet.

What quantization were you using? I've been getting some weird results with 34b quantized to 4 bits -- glitching, dropped tokens, generating Java rather than Python as requested. But 7b, even at 4 bits, works OK. Posted about it earlier on this evening: https://www.gilesthomas.com/2024/02/llm-quantisation-weirdne...

deepseek-coder 6.7b is seriously impressive for how quickly it runs on an M1 Max. There’s a few spots where it still doesn’t fare quite as well as ChatGPT but it’s a small tradeoff considering that it’s fully local and doesn’t even spin up my laptop’s fans.

Have you tried dolphin mistral 7B 2.6?

Part of its training data was code.

No markdown here sadly.

Reply on news.ycombinator.com