Hacker News Reader

Comment by ZeroCool2u | Hacker News Reader

This idea seems to track with the notion we saw early on in the use of ChatGPT where folks suggested that answers with more 'filler' language seemed to produce higher quality responses. The 'pause' token seems like a better way to do this, but it's still pretty bizarre how well it works.

View on HN

It's not that shocking, each token outputted is a fixed 'unit' of computation and so there will necessarily be things which can only be handled if there is more computation. A pause token is one way of allowing the model to spend more time to "consider" the answer.

There seems to already exist a speed up while emitting common strings and a noticeable pause when the next token is more ambiguous. I had attributed it to the model being able to short circuit execution when all probabilities were accumulating in the same bucket.

If you're talking inference speed, it's probably just because simple words are a single token, but complex words or code are much more tokens so it'll output them "slower" (while in reality each token will take the same time)

No, there are significant differences in the per-token time that seem directly correlated with “perplexity” (not necessarily in the exact definition, but so to speak).

Who the heck downvoted this. I’ve literally collected this data myself from my production AI applications. But it doesn’t gel with people’s preconceived notions so they downvote ?

This might be a side effect of speculative decoding.

Models themselves are a fixed amount of computation, but they could be doing more advanced decoding that would run through it multiple times to try to get more truthful answers.

Model does not "consider" answer, it inferes it from training data. If there are "pause" tokens in trainig data, they will be present in inference data with similar density. In other words, "garbage in - garbage out".

I've gotten good results from asking ChatGPT to "check its own work", although on occasion that seems to throw it into a loop and they can't ever get out of it if it doesn't get the right answer, particularly with quantitative answers that it can sanity check itself

Model doesn't "know" "the right" answer. It is statistically guessing, like a bad student with good memory.

Right, but through sufficient statistical guessing, a capacity to solve problems emerges from its ability to use language, which functions well enough as a proxy for some values of "reasoning", for some set of problems, with varying degrees of accuracy.

Or ask it to work step by step

This prompt was a key step forward in the field of human pedagogy as well.

Reply on news.ycombinator.com