Hacker News Reader

Comment by mythz | Hacker News Reader

Qwen2.5 Coder 32B is great for an OSS model, but in my testing (ollama) Sonnet 3.5 yields noticeably better results, a lot more than what the provided benchmarks suggest.

Best thing about it is that it's an OSS model that can be hosted by anyone, resulting in an open competitive market bringing hosting costs down, currently sitting at $0.18/$0.18 M tok/s [1] making it 50x cheaper than Sonnet 3.5 and ~17x cheaper than Haiku 3.5.

[1] https://openrouter.ai/qwen/qwen-2.5-coder-32b-instruct

View on HN

Claude Sonnet 3.5s are bars too high to clear. No other model comes close, with the occasional exception of o1-preview. But o1-preview is always a gamble, your rolls are limited and it will either be the best answer possible from an LLM or it returns after a wild goose chase, having talked itself into a tangled mess of confusion.

I'd personally rank the Qwen2.5 32B model only a little behind GPT4o at worst, and preferable to gemini 1.5 pro 002 (at code only, Gemini is a model that's surprisingly bad at code considering its top class STEM reasoning).

This makes Qwen2.5-coder-32B astounding all considered. It's really quite capable and is finally an accessible model that's useful for real work. I tested it on some linear algebra, discussed pros and cons of a belief propagation based approach to SAT solving, had it implement a fast simple approximate nearest neighbor based on the near orthogonality of random vectors in high dimensions (in OCaml, not perfect with but close enough to useful/easily correctable), simulate execution of a very simple recursive program (also Ocaml) and write a basic post processing shader for Unity. It did really well on each of those tasks.

Not really tried the Claude 3.5, later tried o1-preview on github models and recently Qwen2.5 32B for a prompt to generate a litestar[0] app to manage a wysiwyg content using grapesjs[1] and use pelican[2] to generate static site. It generated very bad code and invented many libraries in import which didn't exist. Cluade was one of the worst code generator, later tried sieve of atkin to generate primes to N and then use miller-rabin test to test each generated prime both using all the cpu core available. Claude completely failed and could never get a correct code without some or the other errors especially using multiprocess, o1-preview got it right in first attempt, Qwen 2.5 32B got it right in 3'rd error fix. In general for some very simple code Claude is correct but when using something new it completely fails, o1-preview performs much better. Give a try to generate some manim community edition visualization using Claude, it generates something not working correct or with errors, o1-preview does much better job.

In most of my test o1-preview performed way better than Claude and Qwen was not that bad either.

[0] https://github.com/litestar-org/litestar

[1] https://grapesjs.com/

[3] https://getpelican.com/

Isn't it a bit unreasonable to compare something free that you can run today on totally standard consumer hardware with *the* state of the art LLM that is probably >= 1T parameters?

Yes

Too bad zed editor doesn't have code completion via custom LLM

So far I am using zed editor and can't switch to somethints else untill zed editor gets the update which support FIM directive via custom LLMs

This question keeps popping up but I don't get it. Everyone and their dog has an OpenAI-compatible API. Why not just serve a local LLM and put api.openai.com 127.0.0.1 in your hosts file?

There is a difference between chat and code completion. While with chat, you can use localhost with llama.cpp, but code completion you cannot do that: https://github.com/zed-industries/zed/issues/12519.

The config for chat, you can do:

  "language_models": {
    "openai": {
      "version": "1",
      "api_url": "http://localhost:8080",
      "low_speed_timeout_in_seconds": 120,
      "available_models": [
        {
          "provider": "openai",
          "name": "Qwen2.5-Coder-7B-Instruct-Q8_0.gguf",
          "display_name": "llama.cpp",
          "max_tokens": 131072
        }
      ]
    }
  },

While for code completion, you have two choices atm: supermaven and copilot: https://zed.dev/docs/completions.

Thank you, makes sense. I haven't used code completion yet.

You could...switch editors? Why not do that until Zed gets that support?

Zed is the best ai code editor with its editable context and super fast renders.

Well, clearly not since it doesn't have those features other editors have.

Yeah, benchmarks are one thing but when you actually interact with the model it becomes clear very fast how "intelligent" the model actually is, by doing or noting small things that other models won't. 3.5 Sonnet v1 was great, v2 is already incredible.

...but you can't run it locally. Not unless you're sitting on some monster metal. It's tiresome when people compare enormous cloud models to tiny little things. They're completely different.

> ...but you can't run it locally. Not unless you're sitting on some monster metal.

I'm getting a very usable ~18 tok/s running it on 2x NVIDIA A4000 (32GB VRAM).

Both GPUs cost less than USD $1,400 on eBay.

qwen2.5-coder:32b is 19GB on ollama [1]

[1] https://ollama.com/library/qwen2.5-coder:32b

I believe that parent comment was referring to the point that Sonnet 3.5 cannot be run locally which it obviously cannot but not because of locally available compute but because it's not OSS.

or probably both? Seriously doubt 3.5 Sonnet is in the same order of magnitude as 32B

Right, it's almost certainly a magnitude larger than 32B.

I'm curious if the M4 Max will be good enough.

That will run Qwen 2.5 Coder 32B just fine. I'm using a M2 Max with 64GB of RAM.

How many tokens per second?

With MLX I got:

    Prompt: 49 tokens, 95.691 tokens-per-sec
    Generation: 723 tokens, 10.016 tokens-per-sec

Yeah, over 65GB VRAM... that'd be expensive but not impossible. I think three RTX 4090's could do it, with their 24GB each.

You really don't need models to fit into GPU's anymore for inference, e.g. llama.cpp is pretty good at partial GPU offload and I've gotten pretty fast results with only being able to fit ~30% of a model in VRAM and the rest in DDR5.

Only 21.3GB is required.

Is this wrong? It says 65.8GB. If it's wrong, what source should I be using instead?

https://llm.extractum.io/model/Qwen%2FQwen2.5-Coder-32B-Inst...

the ollama one is probably quantised.

Ollama's default quantization is q4_0 which is quite bad, but you can go to ollama's model page to see all the quantizations they have available, you can do e.g. "ollama run qwen2.5-coder:32b-instruct-q8_0" which will need ~35G + space for context

... with 4-bit quant and 4000 token context.

Reply on news.ycombinator.com