Hacker News Reader

Comment by thot_experiment | Hacker News Reader

If anyone is looking for a free/local alternative Continue + Ollama is acceptable. If you're just doing run of the mill programming it will work well out of the box.

I'm glad it's open source so I was able to fix most of the issues I had with it and now my copy is in a great place. The documentation is in places many versions behind the actual code so it can be tough to figure out how to set things up when you're venturing off the beaten path. That all being said the granularity of control you have when using local models leads to an experience that's far better than Cursor/Copilot, I really enjoy that it reads my mind a lot of the time now (because I have prompt engineered it to know how I think).

Ultimately, isn't this just the way of things? https://thot-experiment.github.io/forever-problems/?set%20up...

View on HN

Acceptable UX maybe, but the gap between Sonnet 3.5 and open models isn’t worth it. I know people are going to pitch qwen coder 72b, but it’s still a long way off on benchmarks and my time matters more.

You're the second person in this thread to make this point, what are you using it for? I find the difference is basically negligible (in the sense that both get the busywork right and both fail at anything complicated)

yeah, Sonnet goes past that. 300+ line changes in 20 seconds. You have to review it, but generally it's right. It's infinitely faster than the time to look at docs and do it myself.

Sure it's busywork. But it's a lot of busywork very fast.

I gotta find a new trade

Well it's definitely not infinitely faster since you're having to review it, but we're talking about the delta between Sonnet and Qwen/Mistral/Llama or whatever, not doing it manually.

I'm really curious what your problem domain is, like specifically what sort of code are you asking it to change and what changes are you asking for.

I just gave o1 and Sonnet a total layup question (optimization that had a huge win simply by filtering an array before sorting it vs the other way around) and neither model got the solution right, both of them came up with ~hundred lines of code, neither model's code worked on the first try. It took me like 10 minutes to refactor and optimize the code for a 6x speedup and it would take longer than that to debug the AI code to even make it run. (I spent 10 minutes prompting/editing to try to get the generated solutions to run)

Also the initial code was 11 sloc, my solution is 14 sloc, and claude was 70 sloc and o1 was 93. idfk, i just don't think we're there yet

My work involves a lot of boilerplate. Made some changes recently that amount to about 3 lines of meaningful code but wrapped in 4 new files + edits to like 6 others. It's ridiculous. Perfect for AI but I haven't found a way to automate because the tools don't seem to be smart enough to create files and make random little edits, and the amount of words to explain what I want would be too much. One day...

Quick example: create a config class, allowing the caller get and set a variety of config vars. It should be easy to add new vars. Persist any set value to a yaml file. Each var should be described by a name, type, env var fallback value (optional) and default value if the set value and env var are null (optional). The API should be typed and allow int, string, float and lists. Add comprehensive tests.

Obviously nothing complicated but it takes non-zero time. It did it in one shot in about 30s. Didn’t have to look at and docs (I don’t have the yaml lib memorized). Got the python typing right which would have been a bit of a pain. A lot faster than doing it myself, even with reviews. Tests were solid so I could tell it worked.

The filter example you give seems like they should have aced it. Not sure what went wrong but it has easily done work like that for me. I usually am half way through typing the method name when the rest autocompletes. Are you using a tool with good context management like cursor?

On what kind of hardware/GPU are you running that locally?

M1 MacBook Pro is sufficient for a Qwen model.

with how much ram though?

For Qwen-32B, 32Gb should be quite sufficient to run it at 4-bit quantization with full context.

Thanks, I need to figure out something to run on M4 Mac Mini standard 16G

You should be looking at 7-8b sized models, then. https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct is considered pretty strong for its size. That said, you shouldn't expect more than glorified autocompletion at that point.

I'm not familiar with the practical memory requirements on Macs but I suspect that with 16gb of integrated ram you won't have issues running 14B models even at q6_k, certainly would be fine at q4 and it's definitely going to be capable of writing code based on instruction as well as minor refactoring, generating docstrings etc.

The model itself will fit just fine, of course, but you'll also want a large context for coding. And then since it's integrated RAM, it's also used by everything else running on your system - like, say, your IDE, your compiler etc, which are all fairly memory hungry.

Also keep in mind that, even though it's "unified memory", the OS enforces a certain quota for the GPU. If I remember correctly, it's something like 2/3 of the overall RAM.

2x 1080Ti, ~25t/s

I found the plugin did not prompt the model often enough. I would finish writing aline, and go to a new line and sit there waiting to see what the AI thought would be next, only to realize continue wasn't prompting...

Tried a few models and was extremely disappointed by how dumb the completions are. Basically useless. What local ollama-available models do you recommend?

Have you tried Qwen 2.5 coder? I've only done very little testing, but it seemed to work pretty well. I used the 14B version.

https://ollama.com/library/qwen2.5-coder

A software developer's time is much more precious than wasting time on sub-optimal models.

Open Weights models has it's place (in training custom agents and custom services), but if you are knowledge worker, using a model even 5% less than SOTA is extremely dumb

100% disagree with this take, the flexibility in controlling the prompt leads to QwenCoder2.5-32b outperforming gpt-o1 and claude sonnet 3.5 for nearly everything that I use it for (true for Gemma-27b and llama3.3-70b, though in this context I'm almost always using the former). A specialist model that's specifically prompted to do the correct thing will outperform a SOTA generic model with a one size fits all system prompt. This is why small autocomplete models can very obviously outperform larger models at that specific task. I am speaking 100% from experience and ignoring all benchmarks in forming this view btw, so maybe it's just my specific situation.

Also, in general I don't find the difference between SOTA models and local models to be that significant in the real world even when used in the exact same way.

Sounds great.

Does this run with VSCode and how hard is it to set this up?

yes, the vscode extension is a one click install, so is ollama which is a separate project that provides local inference

you'll then have to download a model, which ollama makes very easy. choosing which one will depend on your hardware but the biggest QwenCoder2.5 you can fit is a very solid starting place. it's not ready for your grandma, but it's easy enough that I'd trust a junior dev to be able to get it done

What's the extension name?

Continue, I talk about it at length in the gp post.

Ah, thanks.

I just read the parent post, lol.

Are there any small trained models out there that are specifically for python programming that you know of?

Do you have any example prompts or suggestions for coming up with them?

Reply on news.ycombinator.com