Hacker News Reader

Comment by thatsabadlook | Hacker News Reader

I would pivot to a Chinese provider via open router so fast... Most months on glm or DeepSeek it would be a challenge to spend $100. But now I mostly use local models... Sounds obnoxious bouncing between 2 companies who rug pull their customers every 4 months and degrade service as a form of advertising...

View on HN

> But now I mostly use local models

Do you have beefy hardware? i dream of using a local LLM, but neither my 24GB RAM M4 Air nor my PC with a RTX3080 and 16GB of RAM seem usable (yet).

i could label them usable if "prompt it and then wait for 6 minutes for it to change a single line" is deemed usable, which i don't.

spent the last weekend in a rabbit-hole of which models to use in what kind of setup, but i think with my hardware i'm just out of luck for now.

for private matters I can use Cloud AIs, but not allowed to use it for work-related matters, which is where i could use it the most. they do provide us an isolated AI environment where we can use Claude etc. for work stuff, but heavily rated (about 20 prompts per week per Claude model).

You're super close on your PC. It seems like the sweet spot right now is 24gb vram + lots of RAM for qwen 3.8 flash next or a dgx spark like machine with buckets of unified RAM.

You could run some decent models on your PC l, but nothing that would totally replace serious work. Qwen 3.7 35b is pretty okay. Maybe you could try the new bonsai quant for 3.8 27b but I doubt it would be awesome.

i did in fact test different Qwen's! :) qwen3.5:9b and qwen2.5-coder:14b, i think. but i struggled with speed and them being stuck in "Hmm, wait, no, that doesn't work, let me analyze again"-loops. on the M4 Air I was able to start some bigger models. definitely the first time i burned through 25% of battery in half an hour while getting first-degree burns on my lap.

maybe in 1-2 years my hardware will be enough for fast local-AI. maybe by then i can afford new hardware. (think with current trends option #2 gets less realistic each quarter).

not sure if i lost a bit of curiosity and spark. there's so many models, tools, harnesses, tweaks for harnesses, edited models and so many things. ultimately i don't care enough, it seems vapid to stay bleeding-edge informed about LLMs, if one's career does not directly depend on it. i mean, what the fuck even is a bonsai quant, this stuff makes me feel like an uninformed excel boomer even tho i absolutely don't am one :D

not a software engineer per se so i don't need 16 agents running 24/7 with openclaw. i want a local buddy who helps me write ansible/python faster, better, helps me analyze bugs, refactors small things. basically what Claude Web does, with the benefit of it reading the files itself and running locally.

maybe i just ignore the whole AI/LLM part and just buy some new hardware so i can use higher graphics settings while gaming, WAIT NO, that market was nuked by AI as well, guess i'll continue using my i5 from 2019.

> i mean, what the fuck even is a bonsai quant, this stuff makes me feel like an uninformed excel boomer

Local models and harnesses are the new ~2010s js front end frameworks.

Either accept there's always going to be a new thing of the week and don't over-invest in expecting reliability, or work in another space.

I know it's anathema to "normal" reliable software dev, but that's what new industries look like...

> Either accept there's always going to be a new thing of the week and don't over-invest in expecting reliability

probably have to accept this, yes. wait it out until it's fleshed out.

> or work in another space.

extremely happy to be working in IT, not sure what else i'd do (realistically)

I have a MacBook Air M2 with 24 GB RAM and I run these two Qwen model variations successfully. Try them yourself: * https://huggingface.co/Youssofal/Ternary-Bonsai-2-27B-MTPLX-... * https://huggingface.co/gbuzhf/Ornith-1.5-35B-MTP-UD-APEX-GGU...

So far just testing them for fun in LM Studio. I'm not a coder, so I'm testing them more on logic and language tasks.

But yes, I do have a Mac Studio M3 Ultra with 96 GB RAM where I run the less quantised versions of these for higher quality results: * https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized... * https://huggingface.co/mlx-works/Ornith-1.5-35B-A3B-oQ4e-mtp

Reply on news.ycombinator.com