Hacker News Reader

Comment by gurjeet | Hacker News Reader

I haven't evaluated it yet, but I love the fact that the output is (at least claimed to be) deterministic. I can't trust an LLM to do the right thing after I deploy it to production, because their output is non-deterministic by design.

TERMy (or is it the NPC-forge) seems to be worth a try.

View on HN

You can get determinostic output (mostly) by setting the temperature to zero. Using couple of other tricks you can get close to 100% of determinism with LLMs.

That's reproducible, I wouldn't call it deterministic. Small, semantically meaningless changes in the input can still result in wildly different output.

That's the definition of a chaotic system (small change in initial conditions results in large, seemingly -- but not actually -- random changes in output), but it's still deterministic (same input results in same output).

I've long observed that kind of behaviour in google translate (which makes sense, they have been using ML for a long time.)

> because their output is non-deterministic by design.

It isn't. At least not by design, even though in practice it often can be. If you do greedy decoding (or use a preset seed) and deterministically compute everything (e.g. only use integer math) then it will be 100% always deterministic.

That’s true, but not true-true. Sure, every time you prompt “what is the weather in kansas” you’ll get the same output, but if you prompt “what is the weather in kansas right now” you’ll get a different output, and then “what is the weather in kansas today” gets a different output. Language being language, there are infinite ways to say things, so there are infinite variations in what the llm can output in response to very similar prompts.

This tool has a finite amount of outputs for an infinite amount of inputs. Which is different from an llm based tool.

I think the point being made is that given a particular input string, you can get a deterministic output string back from the LLM.

Yes i think I acknowledged that, but is that useful for making a tool that can be trusted to safely run shell commands when asked arbitrary questions? No. It’s not.

Reply on news.ycombinator.com