Hacker News Reader

Comment by stwrt | Hacker News Reader

The main tradeoff I’ve found with running models like Qwen 3.8 locally is speed. On my hardware, they’re significantly slower than models served through OpenRouter, ChatGPT, or Claude.

That’s been fine for smaller tasks like analyzing a document, but for more involved work like refactoring code the latency makes it harder to iterate.

View on HN

Yeah for live it's a bit on the slower side though still usable on the 4070 ti and 5080 our usual office desk has here, but I found it awesome running on background tasks rather than on demand, things like analyze this and that with this and that ruleset and applicable law warn me with source and justification for any error or missing part etc, qualify each returns with a ranking (certain with source, probable with reason, suggestion),... With its thinking mode enabled and it's great context size abilities it's really powerful, and each employee basically has their own assistant working in the background checking their files on its own as the day progresses

(the main reason is not just cost, it's data not going out and even mess leaving the EU, this essentially frees us of a lot of hurdles)

You find Q3 to be usable quality for your purposes? (I assume Q3 if you're squeezing it into 16 GiB)

Yes, the generic system I built has unsloth/qwen3.8-27b with a 48000 context size running on each employee system with lmstudio, with a simple runner to watch folders and pass it to the llm alongside all context (our company design rules, our legal documents, our contractual rules, our applicable local law and reglementation, ...), return is then mailed to them or added to their personal dashboard event list, depending on their settings.

This is clearly pushing it memory wise and the gpu offloading is only partial but LM studio deal with it automatically and it's being fine even on the 4071 Ti desks (Ryzen 7500F and 32 GB of ram), employees get any feedback in ~20 minutes after they dropped a file (it's much smoother on the 5080 desks obivously), for live it's useless but as background helper it's great and the very large context allows us to fit all the rules we want in there.

One caveat has been to not ask it if everything is ok, but to find what's wrong - but always source and explain it and justify itself, never drown the user in warning in suggestions; goal is to help and provide a second pair of eyes not make them feel annoyed or unsecure.

And I found people to genuinely enjoy something that works for them on their own machine and is not tracked "by the boss", thus the assistant reference, than than a centralized mothership like we also have and they have access to. I also allow them to disable it if they want, I trust them with their work, but a second pair of eyes is always great.

(my previous workhorse for this was Qwen3-14b but it's missing a lot more edge cases)

This is highly helpful to me, thank you!

Reply on news.ycombinator.com