Hacker News Reader

Comment by jonplackett | Hacker News Reader

I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.

It was actually great. I have like a non-AI box so to speak 8GB VRAM, co-incidentally from a gaming PC ...

All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period.

Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev`. while any other model previously just entirely failed to create anything usable even with actual guidance.

Since then I have actually gotten a gemma-4 12B qat 4bit quantized with a ~250MB MTP from unsloth to work with a 32k context "working" on this setup at 80-120 t/s. That's usable for private stuff on a co-incidental box!

It's still only 32k context and it's entirely dumb vs. our API paid at-work Claude Opus. But for entirely private local stuff it's totally workable without breaking the bank even after all these AI price hikes!. I bought this rig literally just for gaming a month ago.

Have you tried a mixture of experts model? Dense models have been quite slow for me, as I have only 6 GB VRAM. But with llama.cpp and --cpu-moe I get 200 t/s input and almost 30 t/s output with Gemma 4 26B A3B, which feels ok to use. Would be interested about your mileage there.

I wish qwen3.8 had a MoE variant, but the skinny is it won't be coming.

I'm having the same constraint, 8GB VRAM, and was trying the past months to get a local model running to drive my Home Assistant setup via voice, so many tool calls, little room for errors. I've tried multiple models, different temperature and top p values, different context windows... turns out that a quantized gemma4 is the best pick so far but qwen3.5:9b isn't far off. Surprisingly newer models are not necessarily performing better.

I've had great success using Gemma4 even on old hardware (4GB VRAM). I haven't found a Qwen model that can match it on very low end hardware.

That's funny, I downloaded the same model on my 48GB M4 Pro and gave it a problem to solve in an existing codebase, it spun its wheels for twenty minutes and then fell over dead. This was using LMStudio and pi as a harness; I never use pi for anything else, so maybe I'm holding it wrong.

its all still somewhat of a dice roll

We don’t know what quantization level was used for the weights or the kv cache for you or for parent poster, so this is probably an apples to oranges comparison.

They made a kind of strange decision with Qwen3.8 27B, the template defaults the reasoning_effort to xhigh. I found if you set it to medium it doesn’t just sit there churning forever.

xhigh gives better results

Not necessarily.

I have seen xhigh go down several rabbit holes, dwell on edge cases and write worse code as a result; it literally distracted itself into writing a complex chain of functions ignoring my prompt, when on “low” reasoning it gets it right on a prompt that requires a few lines of code in the right places.

Simon Willison’s blog has another example (SVG of a circle).

It’s a bit like how giving LLMs access to web search tools can cause them to go down a blind alley based on their first “reasoning” output that then leaves them unable to solve a puzzle correctly that they can fully solve on their own.

With qwen 27b, setting the right reasoning effort for the specific task is important. With xhigh it has a chance at hard problems that bigger models may even fail. But for many everyday tasks, I have found that no reasoning and a system prompt instructing it to be brief is good enough. Note that even with thinking disabled, it may still get into long "chain of thought" reasoning state (out of thinking blocks) if the task is hard and you do not give further instructions, esp with access to tools etc.

Not if it fills up its entire context with "But wait..."

Is there an easy way for a n00b with LMStudio to switch it to medium? Asking for a friend… XD

I had heard of this and actually did set the reasoning to medium ahead of time…

I've recently learned and then observed that oMLX serves local models much, much faster than LM Studio.

Set its thinking lower. This is a known issue. It still thinks A LOT with lower reasoning levels

Maybe giving pi more output by setting higher value to maxTokens will resolve his issue

I got the qwen 3.8 abliterated model running on my MacBook Pro M5 48GB and it's pretty nice having a local model that can do a lot of experimentation without rails.

orcarouter or obliteratus?

My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?

Mineral oil bath?

just decent air cooling and you'll be okay. it will get up to 75/80C though for my M5 MBP.

I don't have the hardware but a often mentioned advice is to put your mac into energy saving mode - it still will work, a bit slower, but stays cool.

Imagine spending all that money on Apple hardware only to throttle it to a fraction of its performance lol

Steve Jobs would be proud. People really are holding their Apple hardware wrong.

It’s hot and also LOUD and runs the battery down quick.

But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in.

It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever?

I have an m5 max 64gb and I think anything slower would be quite painful.

I have an m3 max 64GB and it runs ok, I was playing around with it last night. Its reasoning is better, but the token/s still makes an MoE model (35B A3B) preferable so I switched back.

Try using Goose instead of Claude's harness? Goose doesn't load as much context in at the start, so it might be more usable. I've definitely been able to get it past 128K, although I typically don't go beyond 70-80k. MoE might also be better at this.

>It’s hot and also LOUD and runs the battery down quick.

Who'd run this on battery?

Or do you mean kills the battery faster even when used while charging?

There was a lovely window of a few years when processors were fast enough and low-power enough that real development work could trivially happen on a Macbook Air in a lounge.

That window is waning as more and more memory and graphics processing power is being used locally.

I suppose the future is most likely going to involve farming out AI requests to your desktop machine, your company's compute farm, or a cloud endpoint, but if you're expecting to run an AI offline on your lap with a battery...your lap will get warm.

No idea what you are talking about. My battery lasts longer than ever while running vim and make and GCC. It’s amazing. Not sure why your windows are closing.

Because the local LLM, which you are not running, is running for much longer than gcc and is eating the battery. Different choices, different outcomes.

So don’t run a local LLM?

Bur the implication here in the parent comment is that for ‘real dev work’ to be done you must be running an LLM, and therefore you can’t do it on a MacBook anymore.

Turns out you can still accomplish stuff with a text editor and compiler.

Of course we can code without a LLM, but if we want to, as the sibling comment says, don't run a local LLM on a laptop. I'm offloading the LLM to the LLM vendor, like most of us IMO. The CPU is still busier then it used to be, because of the local harness.

I envision a future when the potential energy in a laptop battery, could take out 17 city blocks if it exploes.

Running your dev environment on your laptop is so 2025.

Running the laptop hot ages the battery even when you’re plugged in.

Merely existing noticeably ages most battery chemistries.

You can take half an hour out of every week inconveniencing yourself to protect your battery, or you can spend an hour once a year just putting in a new one (or having it done for you in-store if you're using an Apple device) and save yourself the nagging worry.

I was trying to do some work on the train with no signal.

Couldn't you just enjoy the ride?

I remember doing this with crypto mining way back in the day. That machine will die within a year.

If you're running it while idle and don't need the quickest results, reducing the clock speed improves energy efficiency (and in your case avoids overheating the battery). There will be some optimal speed that maximizes computations per joule that depends on the specific load and can only be found by measurement.

On Linux, you can cap CPU frequencies with "cpupower". Does MacOS have any equivalent?

The Mac unfortunately just has two performance modes ‘all out power and melting’ or ‘cold and really really slow’

My M5 Max has 3 modes, "low", "automatic", and "high". Automatic doesn't seem to simply switch between low and high -- it seems to sit in the middle and vary dynamically based on workload and temperature. With automatic I get more than half as many tokens per second as on high, but with much lower fan speed and temperature.

I tested Gemma 4 a couple months ago and yes it ramps the fans to max. Definitely a thing that should only be done on wall power.

Macs fan control set to full blast and lifting the Mac above table for air circulation across bottom.

I use M1 Max with qwen3.8 27b mlx. The gpu temp can reach easily to 95°C as fan doesn’t kick in automatically until 90°C. With Macs fan control at full blast and MBP off table, temp usually hover around 85°C.

Just curious, what is your ram and how many tokens per second do you get with that?

I am getting like ~25 tps with M5 Max 64GB. Unsloth Desktop and Qwen 3.8 27B Q6 KXL

RAM 32GB, tokens are less than 20, may be about 12-15. Main issue is context window, I have set to 64k but practically beyond 32k, too much swap and slow down.

Yikes isn’t 95 well above the danger zone for most components or have things changed? I remember crypto mining in the early 2010’s and making sure to stay under 85C on my GPU.

Stuff runs so hot these days. I think one of the parameters pushed to keep Moores law from completely flattening was heat. My desktop regularly hits 95 just compiling code.

I guess thermal throttling will inevitably kick in.

If you're going to try to use a mac laptop for long term inference you'll probably want to buy one of those $30 laptop fan cooling pads (sold for absurd sized 17 inch x86-64 gaming laptops) on Amazon. Otherwise you'll roast the battery over time and significantly hurt its cycle life and capacity. If you can get significant airflow under the laptop that'll help a lot.

Having a server in the basement helps a lot :-D Then tailscale from everywhere.

Which exact model are you running? With only 48GB of RAM, by the time I got a model small enough, it was pretty bad in performance both in speed and reasoning.

How many tok/s are you getting? What gen mbp?

i suspect ppl dropping generic "its awesome" comments are not actually using it and prbly just managed to get it running for a prompt or two.

Yeah, that's my experience. It's a big "wow" factor to get a non-trivial LLM running on my Mac, but it's actually not that useful. Like trying to use Photoshop at 8 FPS.

Regarding this analogy, fps don't matter as much for Photoshop, since it's not an immediate mode GUI. 8 fps would be quite ok for comfortably getting feedback on live image filters and such.

It’s not because you didn’t find use cases for local LLMs that there are none.

I use local LLMs on my Mac Mini M4 Pro with 48G to review text messages tone, act as a text correction tool, act as a code review tool, to do code agent work, generate code snippets, etc

Gemma 4 26B A4B gives me steady 20 tps.

I feel like it's 50/50 between people doing that, and people that have spent a lot of time tuning a system they are pointing at focused and well specified problems.

Seems threads about local LLMs on Apple hardware feature comments listing M3/4/5 at 48GB 64GB and not 128GB.

That is, users with M-series hardware that have less-than-max RAM share results whereas users with max RAM do not.

Speculating (not extrapolating), maybe users with machine that have max RAM are less interested in running local LLMs and are less averse to paying services for compute?

Personally, I’d love to see what output max RAM M-series Apple hardware in these threads.

Qwen3.8-27B runs at 59.5 tok/s on my M4 Max, 40-core GPU, 128 GB

I use it occasionally for classification and other tasks but I wouldn't trust those smaller models with the real work and for larger data processing it's too slow, e.g. a dataset I wanted to classify would've taken 56 days on my laptop vs just paying the cheap Luna prices to openai and getting it done in a few hours.

Not sure I would trust Luna with that. Deepseek Pro Max and Code Mode I would be more inclined to trust.

I've been using Qwen3.6-37B-A3B on an M1 Max w/ llama.cpp and for my practical uses I prefer it to qwen3.8. When 3.8 does answer its slower and, qualitatively, marginally better than qwen3.6, but 3.8 often ends up in unresolved thought loops and runs slower. The Moe 3.6 on my setup is much faster, 500t/s peaks, 30t/s typical, vs 3.8 150 peak, 4-9 t/s typical.

While I've spend a little time tuning, I'm assuming there will be deeper tuning for 3.8 that might close the gap.

My M4 Pro 48GB gets about 13tok/s, in both 3.6 and 3.8 27b Qwens. Qwen A3B and Gemma get closer to 100tok/s from memory but the results are pretty poor for coding tasks.

Edited to add: for agentic workflow I’m running omlx which tells me it has about a 90% cache hit rate (tradeoff is some disk and mem space) - that noticeably changes the felt speed.

I get around 20 tok/s, 4 bit quant, MTP, 4 bit KV cache quantisation. On an M4 Pro 48Gb.

M3 Pro 36GB. I am getting 17 tps with MTPLX.

We were trying running a local gpt-oss 80GB model on a H100, and honestly I was surprised how dumb it was.

gpt-oss is about a year older than qwen 3.8 27b

GPT-OSS 20B didn’t really merit the fanfare even when it was released; it’s definitely not competitive now. Even the 120B version has been well eclipsed by smaller LLMs at this point. The last version of Qwen 27B/35B was better, and now the new one is even better than that!

Was there a more recent refresh or is this the model from a year ago? The frontier models were barely functional and almost useless a year ago (gpt oss was pre opus 4.5!) - I would be very surprised if the original drop is anything more than totally obsolete/irrelevant at this point

Yes, the old entirely stupid old gpt-oss. But Sonnet and GPT were very useful then already, qwen also.

How much RAM? And what do you use it for if I might ask?

how quick does it respond? what are specs of your laptop?

Does it need to respond fast? For important applications, I'm sure we'd all be fine waiting 20 minutes for a high quality, usable answer. Or is it the need for interative refinements that make speed relevant?

It requires patience but it’s more like waiting 5 mins for it to do tasks. You need to be much more involved though and do things slower than Claude where you can trust it to do a lot of tasks at once. It doesn’t have the context for that

if you are so sure about what the final shape of your output is then its prbly not a common use of ai

If you are sure about the final shape of your output it's a great use case for AI, as you can define what you want in your prompt and refine towards it!

It's where you don't know the end state you're looking for that you'll end up generating slop on top of slop and creating a whole Gastown just to power your Gastown.

I tried it on my M1 MacBook Pro. It's slow but surprisingly smart as a general purpose LLM. Maybe GPT-5.3 level. I gave it a bunch of tools and it can search the internet, make product recommendations, document, code, etc.

Had the same reaction so had Grok create a script to:

- find a free GPU droplet on digital ocean

- fire it up

- pull in a snapshot of the model + extra files/packages etc

- set up a ssh tunnel so that the localhost:8000 routes to the above

Then I just configured OpenCode to use the above and was off to the races.

Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could get expensive with 24/7 running)

Awesome, I used Claude to write a small python script to do the same with Linode's API. The only difference is I setup a persistent drive, and with Linode you can boot off of it. So my biggest start up lag is ~ 2 minutes to deploy + boot, then maybe 2 more to warm the model.

I actually dislike LLMs. But I'm a realist, and on-demand compute like this is massive cost saving measure.

(persistent drives are relatively cheap, compared to a box with several GPUs.. or even one. I find it worth the expense)

> the only difference is I setup a persistent drive,

Going to try this out vs the snapshot!

I also really like this experiment b/c it's a mix of LLMs and old school IaaC/DevOps.

> Works out to be about ~$2/hr all said

What GPU you end up with for that price? Vast.ai (https://cloud.vast.ai/?priceInstanceHourlyMax=2) has a bunch of setups available to reach 192GB VRAM under $2 :) Quick skim showed 4x48, 2x96 and 8x24, all for under 2 buckaroos or around there.

It's actually a bit less than $2 but I rounded up just to make the math easier.

And yeah, did a lot of work with Vast AI at a past job and it's pretty wild the variety of prices/hardware that they have.

> It's actually a bit less than $2 but I rounded up just to make the math easier.

That's cool, what actual GPU though? I'm still curious :P

Did you read even the title?

Reread what he said maybe?

Reply on news.ycombinator.com