Hacker News Reader

Comment by SyneRyder | Hacker News Reader

TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally.

I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.

View on HN

Thanks for the feedback. I wanted to get into more detail, but I spent the whole weekend working these problems and then constructing this post.

Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.

Understood, and I realized you'd posted this to HN yourself, so I felt a bit bad making the comment. I think maybe for me, this might have worked better if the motivation had been one separate post, and the details of the gotchas as a post of its own.

But I also think, if you're going to be limited to 65k token windows, you're going to have a really difficult time. Even 250k windows were cramped for me when that's all we had on Anthropic models. I just don't think a 65k window is going to be big enough for proper cyberdefence work, even if I totally agree with going local wherever you can. It feels like if you're defending against swarms of 1-10M context windows, you need to get as close as you can to similar. I've had to reach for Chinese 1m models instead because the American models just refuse me here in Australia.

Well, if you are serious about it and you have Strix Halo, there are better ways of getting more context and capability and speed. Lookup halogen for Strix The most cost-effective local option right now, I think, is dual R9700. You can run a 27B dense Qwen at FP8 around with a full context and 2-3 concurrent sessions of 260K context. If you go down to an MXFP4, you get 4 to 5 concurrent sessions. And speed is on par with anything you'll get from hosted providers. You're getting between 60-80 for FP8 and 150+ tokens per second speed for MXFP4 and pp is 4K+. Lookup vllm radiance There is also a lot of progress in running Qwen 3.8 next flash with dual R9700. Obviously one gets less context and speed is a little bit less, but it's still very acceptable. Better than what you're getting with Llama on Strix Halo, that's for sure.

These are small dense models,meaning Qwen, have gotten capable and fast. And there's been a lot of progress in the area. So sticking with Llama you are not taking advantage of the hardware you have. And yeah, for Strix Halo, you should just look into halogen and you shouldn't be just using 96 gigs for the VRAM. You should give it most of the VRAM to the inference and connect to it from your laptop or something. People ar egetting 1000+ pp with qwen 3.8 Next Flash

Thank you. This is the hackernews I miss.

Your hardware can do way more than 64k tokens context window, can't it? And with Ollama it's very easy, superficially you just drag the slider.

I'm now reading "Friends Don't Let Friends Use Ollama" linked in another comment so a lot of problems with that approach are surfacing for me right now.

So yeah. Along with others, I think you should come up with some empirical means of understanding if your preprompt is doing anything good since I doubt that it's all necessary and helpful. Second maybe you and I need to fix our runtimes.

Yes.

I have to observe:

Seeing the raw CoT is the best signal for evaluating prompt efficiency. The fact that frontier providers don’t share this feels criminal after this last weekend. They’ve played us all for fools.

Thanks for the token setting observation. I’m running ollama headless and from the command line. I didn’t even know there was a web ui. Conf file is pretty straightforward though.

I settled on 64k tokens because of a recommendation from an LLM. I now see that that guidance is associated with similar sounding (but not the same) configuration constraints. So this weekend I’ll try turning it to 11 (250k tokens?). Anticipating some hard locks over here.

CoT is performative and doesn’t reveal how reasoning happens. If you look at those traces locally it’s just gibberish, especially if the model falls into a loop.

https://arxiv.org/abs/2605.11746

Yes, please!

The new DeepSeek models address this issue very cleanly. DeepSeek Flash V4.1 requires less than 1 GB memory for a full 1M context, down from about ~10 GB in DeepSeek Flash V4.0. This is a significant step towards making near-frontier models usable even on low-end consumer hardware, though of course with significant tradeoffs in overall performance.

Reply on news.ycombinator.com