Hacker News Reader

Comment by thot_experiment | Hacker News Reader

Running it locally lets you INTERJECT IN IT'S THINKING IN REALTIME and I cannot stress enough how useful that is.

Interesting.. In the official API [1], there's no way to prefill the reasoning_content:

> Please note that if the reasoning_content field is included in the sequence of input messages, the API will return a 400 error. Therefore, you should remove the reasoning_content field from the API response before making the API request

So the best I can do is pass the reasoning as part of the context (which means starting over from the beginning).

[1] https://api-docs.deepseek.com/guides/reasoning_model

You mean it reacts to you writing something while it's thinking of that you can stop it while it's thinking?

You can stop it at any time, then modify what it's written so far...then press continue and let it continue thinking and answering.

Fundamentally the UI is up to you, I have a "typing-pauses-inference-and-starts-gaslighting" feature in my homebrew frontend, but in OpenWebUI/Sillytavern you can just pause it and edit the chain of thought and then have it continue from the edit.

That's a great idea. In your frontend, do you write in the same text entry field as the bot? I use oobabooga/text-generation-webui and I findit's a little awkward to edit the bot responses.

No, but the chat divs are all contenteditable.

Oh! That is an excellent solution. I wish it was that easy in every UI.

Thanks, for what it's worth unless you particularly need to use exl2 ollama works great for local inference and you can prompt together a half decent chat UI for yourself in a matter of minutes these days which gives you full control over everything. I also lean a lot on https://www.npmjs.com/package/amallo which is a api wrapper i wrote for ollama which makes this sort of hacking very very easy. (not that the default lib is bad, i just didn't like the ergonomics)

Oh this is so cool

How are you running it locally??

I am running a 4bit imatrix quant of the 70b distill with quantized context. It fits in the 43gb of vram I have.

Reply on news.ycombinator.com