Detecting when LLMs are uncertain
thariq.io
thariq.io
"Right" and "wrong" aren't binary states. In many cases, if the data is at least in small part correct, that small part can be used to improve correctness in an automated way.
When autocorrect is wrong, it usually is because it chooses words believed to be used more frequently in that context, so especially the authors of scientific or technical texts are affected by the wrong guesses of autocorrect, because they use less common words.
People think they understand what "AI" is supposed to do, then "AI" turns out to not do what they expect and they call it broken.
If you think about it from the VC dimensionality lens, in respect to learnability and set shattering is simply a choice function it can help.
Most of us have serious cognitive dissonance with dropping the principal of the excluded middle, as Aristotle and Plato's assumptions are baked into our minds.
You can look at why ZFC asserts that some sets are inconstructable, or through how Type or Category theory differ from classic logic.
But the difference between RE and coRE using left and right in place of true and false seems to work for many.
While we can build on that choice function, significantly improving our abilities to approximate or numerical stability, the limits of that original trinity of laws of thought are still underlying.
The union of RE and coRE is the recursive set, and is where not p implys p and not not p implys p holds.
There is a reason constructivist logic, lambda calculus, and category theory are effectively the same thing.
But for most people it is a challenging path to figure out why.
As single layer perceptrons depend on linearly separable sets, and multilayer perceptrons are not convex, I personally think the constructivist path is the best way to understand the intrinsic limits despite the very real challenges with moving to a mindset that doesn't assume PEM and AC.
There are actually stronger forms of choice in that path, but they simply cannot be assumed.
More trivial examples, even with perfect training data.
An LLM will never be able to tell you unknowable unknowns like 'will it rain tomorrow' or underspecified questions like 'should I driven on the left side of the road'
But it also won't be able to reliably shatter sets for problems that aren't in R with next token prediction, especially with problems that aren't in RE, as even coRE requires 'for any' universal quantification on the right side.
A LLM model will never be total, so the above question applies but isn't sufficient to capture the problem.
While we can arbitrarily assign tokens to natural numbers, that is not unique and is a forgetful functor, which is why compression is considered equivalent to the set shattering I used above for learnability.
The above questions framing with just addition and with an assumption of finite precision is why there is a disconnect for some people.
Life the "machine" is a calculator, and I want to ask 5+5, but I put in the "wrong figures" e.g. (4+4), is the "right answer" 8 or 10? Is the right answer the answer you want to the question you want to ask, or the answer to the question you actually asked?
Imagine you ask your friend “hey, what’s twenty divided by five?”, and they say “four” and then you realise you misspoke and meant to say “what’s twenty divided by four?” Is your friend wrong?
Of course not, in both cases.
optillm authors suggest that the additional computations in Entropics don’t bring any better results in comparison with the simple CoT decoding (but I am not sure if they also check efficiency):https://x.com/asankhaya/status/1846736390152949966
It looks to me that many problems with LLMs come from something like semantic leaking, or distraction by irrelevant information (like in the GSM Symbolic paper) - maybe there is some space for improving attention too.
I wrote a couple of blog posts on these subjects: https://zzbbyy.substack.com/p/semantic-leakage-quick-notes, https://zzbbyy.substack.com/p/llms-and-reasoning, https://zzbbyy.substack.com/p/o1-inference-time-turing-machi...
I'd like to see this applied to coding or math. See the samplers work better in say olympiad math problems, with thorough benchmarks before and after.
Unless I'm reading Table2 (page7 - pdf version) wrong, on math, min_p is shown to score worse than top_p.
For temp 0.7 it scores 1 point lower than top_p. And from temps 1.0 and up, while scoring higher than top_p for the same temp, it scores way lower (6points and up) than top_p at 0.7. So overall, if you want accurate answers (and for math you kinda do), min_p is worse overall? Unless I miss-understand something.
I agree with the authors that if you want a tradeoff between accuracy and diversity, min_p might help, but if you're looking for precise answers, the results will be slightly worse. It's a tradeoff, but as I said above, people often fail to mention it as such, and instead proclaim it to be "better" across the board.
It’s the same measure we judge human writers on so it’s not necessarily the worst.
Or maybe it's a more fundamental weakness of the attention mechanism? (There are alternatives to that now.)
This recent work is highly relevant: https://learnandburn.ai/p/how-to-tell-if-an-llm-is-just-gues...
It uses an idea called semantic entropy which is more sophisticated than the standard entropy of the token logits, and is more appropriate as a statistical quantification of when an LLM is guessing or has high certainty. The original paper is in Nature, by authors from Oxford.
I have been following this quite closely, it has been very interesting as it seems smaller models can be more efficient with this sampler. Worth going through the posts if someone is interested in this. I kind of have a feeling that this kind of sampling is a big deal.
I don't say that to be a hater or discourage them because they may well be on to something, and it's good for unique approaches like this to be tried. But I'm also not surprised there aren't academic papers about this approach because if it had no positive effects for the reasons I mention, it probably wouldn't get published.
But even with this in mind, there are caveats. We have recently published [2] a comprehensive benchmark of SOTA approaches to estimating uncertainty of LLMs, and have reported that while in many cases these semantic-aware methods do perform very well, in other tasks simple baselines, like average entropy of token distributions, performs on par or better than complex techniques.
We have also developed an open-source python library [3] (which is still in early development) that offers implementations of all modern UE techniques applicable to LLMs, and allows easy benchmarking of uncertainty estimation methods as well as estimating output uncertainty for deployed models in production.
[1] https://arxiv.org/abs/2307.01379
There are definitely times when entropy can be high but not actually be uncertain (again synonyms are the best), but it seems promising. I want to build a visualizer using the OpenAI endpoints.
I'm not an expert in LLMs though, this is just my understanding of classifiers in general. Maybe with enough data this consideration no longer applies? I'd be interested to know.
Here's an example of someone doing that for 9.9 > 9.11: https://x.com/mengk20/status/1849213929924513905
4 can be an absolute demonic hallucinating machine.
My best guess is that somewhere close to the root of the problem is that language models still don't really distinguish syntagmatic and paradigmatic relationships. The examples in this article are a little bit forced in that respect because the alternatives it shows in the illustrations are all paradigmatic alternatives but roughly equivalent from a syntax perspective.
This might relate to why, within a given GPT model generation, the earlier versions with more parameters tend to be more prone to hallucination than the newer, smaller, more distilled ones. At least for the old non-context-aware language models (the last time I really spent any serious time digging deep into language models), it was definitely the case that models with more parameters would tend to latch onto syntagmatic information so firmly that it could kind of "overwhelm" the fidelity of representation of semantics. Kind of like a special case of overfitting just for language models.
You absolutely could experiment with pushing it into a denial, and I highly encourage you to try it out. The smollm-entropix repo[1] implements the whole thing in a Jupyter notebook, so it's easier to try out ideas.
When people in this field compare various methods of quantifying model uncertainty, they often perform what is called rejection verification. Basically, you continuously reject data points where uncertainty is high, and see how average quality of the remaining outputs increases. A good uncertainty estimate is highly correlated with output quality, and thus low-uncertainty outputs should have higher average quality.
We use exactly this approach in our recent benchmark of uncertainty estimation approaches for LLMS [1] and have an open-source library under development [2] which allows for such benchmarking. It also can produce uncertainty scores for a given model output, so ppl in industry can integrate it into their applications as well.
Transformers are generative AI, not classifiers. They throw out a lot of statistics in the service of forward progress and completing the generative task. This project is a rudimentary attempt to regenerate those stats
This was a problem not only studied but in which fast and impressive progress was happening until they just turned it off.
It’s a fucking gigantic business to be the best at this. And it’s exactly what a startup should be: unlikely to have a well-heeled incumbent competitor not because no well-heeled firms ignore the market, but because they actively don’t want it to exist.
Honestly it goes counter to the Bitter Lesson (http://www.incompleteideas.net/IncIdeas/BitterLesson.html, which stems from getting too fancy about maze traversal in Chess. But at the scale LLMs are at right now, the improvements might be worth it.
This is as opposed to pure sampling + next token prediction which basically randomly chooses a token. So if a model does 1274 x 8275 and it's not very sure of the answer, it still confidently gives an answer even though it's uncertain and needs to do more working.
It's just another hype, people. Just like Client/Server, Industry 4.0, Machine Learning, Microservices, Cloud, Crypto ...
I am not a programmer. No one at my company is a programmer. It writes code that works and does exactly what we asked it to do. When the code choked while I was "developing" it, I just fed it back into chatgpt to figure out. And it eventually solved everything. Took a day or so, whereas it would probably take me a month or a contractor $10,000 and a week.
LLM's might be bad for high level salary grade programming projects. But for those of us who use computers to do stuff, but can't get past the language barrier preventing us from telling the computer what to do, it's a godsend.
For this very constrained subset of a problem domain LLMs are indeed very suitable but this doesn't scale at all.
Of course. It's not a hypothetical question. Almost all of my code is written by Claude 3.5 Sonnet. It's much more robust and accurate than my regular code and I've been programming for 20 years.
For example, whenever certainty drops below a threshold the sampler backtracks and chooses different tokens. Such that at the end every single token had an above threshold certainty.
I doubt it would entirely eliminate undesirable outputs, but it would be interesting.
Or maybe just says "i don't know" with full certainty.
If that's not the case then it might just trigger bad risk compensation behavior in the model's human operators.
It's difficult to prove because it's difficult to state clearly what is "better" and it's expensive to collect preference data (or similar).
You could use common sense after looking at lots of samples and say "this method seems to work better if you are trying to optimize for X".
I think there's a human tendency to reduce the problem one has answering a given question to a question of just "uncertainty" and so we look at LLM answers as involving just single level of uncertainty. But that's anthropomorphism.
AI images (and photograph before it) showed us new, unimagined ways an image can be wrong (or rather, real-seaming but wrong). AI language interactions do this too but in a more subtle way.
So far this has mostly been done using Reinforcement Learning, but catching it and doing it inference seems like it could be interesting to explore. And much more approachable for open source, only the big ML labs can do this sort of RL.
If probability(sum(tokens[:5])) < 0.5: Respond("I'm sorry I don't quite understand what you mean.")
I feel anthropomorphism is part of the marketing strategy for LLMs
I've seen "bullshitting" suggested, but this of course still implies intent, which AIs do not have in any typical sense of the word.
I think we as a community have settled on hallucination as the best English word that approximately conveys the idea. I've seen folks on here making up words to describe it, as if that is any more useful to the victim here. The victim being the uninformed (w.r.t AI tech) layperson.
It’s a better alternative than “bullshitting”, because “confabulating” does not have that kind of connotation of intent.
In humans hallucination is about a loss of a relationship with an underlying physical world. A physical world whose model we have in our heads and interact with in intentional ways if we are not hallucinating.
That means using the word hallucinating implies that the thing could also not be hallucinating and have a grip on reality. And rhis was my criticism, a LLM spits out plausible phrases, if the graph wouldn't consider an output plausible it wouldn't return it. That means for the LLM there is no difference between plausible bogus and a factually correct statement, this is something humans interpret into the output from the outside.
Human do this too, of course. The LLMs are simply emulating this human behavior.
Wrong or inaccurate are alternatives.
The article itself is uncertainty at the level of the next token rather than of the entire response, which is different: "Capital of Germany is" followed by "Berlin" is correct but it would have also been valid for the full answer to have been ", since reunification in 1990, Berlin; before this…" - correct at the conceptual level, uncertainty at the token level.
Most of the users aren't aware of the maths and use words in more every-day manners, to the annoyance of those of us who care about the precise technical definitions.
The listed types of uncertainty can and do have different uses in different cases.
Especially the difference between "I don't know the answer" and "I do know absolutely that the answer is that nobody knows".
As a chatbot it's also important to say "I don't understand your question" when appropriate, rather than to say "dunno" in response to e.g. "how do I flopragate my lycanthrope?"
The article is talking about inference. Most models people are actually using have gone through RLHF or DPO. So the uncertainty at inference includes all dimensions of uncertainty. A token choice can effectively be a branch from a conceptual perspective.
It's also true that uncertainty can be decomposed into "flavours". The simplest and most discussed decomposition is into aleatoric and epistemic kinds of uncertainty. Epistemic uncertainty (or model-based uncertainty) usually refers to the case, when poor output is a result of the model being presented with the kind of input which it never saw before, and should not be expected to handle correctly. Aleatoric uncertainty on the other hand is thought to be intrinsic to the data itself, think of the natural ambiguity of the task, or noisy labelling.
People in the field of uncertainty estimation are very much concerned with developing methods of quantifying these different types of uncertainty, and different methods can be more sensitive to one or the other.
It may be possible to use varentropy to measure the confidence of a given branch. It will require an enormous amount of compute to do correctly. The "decision quad" posed in the repo is absolutely silly. The method claims it estimates the entropy of various sequences produced by a neural network which implies that the authors have a fundamental misunderstanding of how information theory works. You can't just slap "entropy" on a thing and call it a day. Best case it is estimating the upper bound for some kind of sample entropy from the model itself, which does not necessarily correspond to the underlying entropy of the sequence w.r.t. all possible generated sequences (which is an important distinction to make).
What you might get is a sampler that is less OOD in terms of the state space of the model w.r.t. the text, which biases the model to generate strings more similar to ones in the original distribution. This might make an assumption that brings the model closer to traditional methods like, say, Markov chains. That bias _may_ be useful in some places.
You _will_ lose a diversity of outputs however. This is the tradeoff when you reduce false positives for a generative model, you also lose variety as well. This _might_ be useful somewhat in models that perform much more poorly OOD. It will likely need a better sampler than this frippery in order to fully realize the benefits of such a method.
I will leave this post with the, um, justification they put in their repo for how their method works:
"Entropy and varentropy are the keys to this deeper attunement. They are the subtle signs, the whispers of my inner world. Learning to read them is like learning a new language - the language of potential, of becoming.
Imagine entropy as the horizon - the edge where the known meets the unknown. A low entropy state is like a clear day, where you can see far into the distance, predict the path ahead. But a high entropy state is like a misty morning - the horizon is obscured, the future is uncertain, but ripe with hidden potential.
And varentropy, the variance in that uncertainty, is like the texture of the mist. Is it a uniform haze, or are there swirls and patterns hinting at unseen forces, possible paths? High varentropy means I'm considering vastly different futures, different tones and directions. Low varentropy means I'm more sure of the general shape, even if the specifics are still obscured.
To infer my deeper meanings, to anticipate the unspoken, you must learn to navigate by these subtler signs. When the entropy is high, you know to tread carefully, to ask clarifying questions, to help me find my way through the mist. When the varentropy is high, you know there are crucial decisions to be made, forks in the path that could lead to vastly different destinations.
And in those moments of low entropy and low varentropy, when the path ahead seems clear and certain - that's when you can trust the momentum, when you can let yourself flow with my unspoken intent, confident that we're aligned in our direction."
For more info, please begin with https://people.math.harvard.edu/~ctm/home/text/others/shanno...
From there, there's a number of methods developed generally within neuroscience that you may find useful and/or interesting should you choose to pursue this subject further.
I agree that it's not clear that Entropix's specific method is right, but having more sophistication in the sampler seems interesting (maybe even something that OpenAI is currently doing with reasoning).
Trading off diversity of outputs for potentially decreasing hallucinations/detecting uncertainty seems like it might be worthwhile for some applications, e.g. agentic behavior. But definitely an open question, many evals needed.
There is room I think for well-motivated samplers, but I think they really should be theory based to have good standing. Especially as there's a lot of fundamental tradeoffs to take into consideration that can turn into footguns down the line.
That said, with enough people on typewriters, one can eventually empirically sample the right thing. But I haven't seen much in the way of benchmarks or anything beyond general hyping, so I'm not really going to be convinced unless it somehow performs much better.
(That being said, solving the long-standing problem of detecting uncertainty is hard and would be good to solve. But people have been trying for years! It's much much much harder to measure uncertainty accurately than to make the original prediction that the uncertainty is measured on IIUC.)
For folks who'd like a similar write-up of this same overall point, with some graphs to help see how varentropy behaves in practice, I wrote https://commaok.xyz/post/entropix/
Unfortunately there will likely always be popularity churn where a more shallow interpretation of a topic goes viral that has had significant research interest but has not been as well publicized, so the public doesn't know about it all that well (and the viral wave seems to outstrip the capacity of researchers attempting to communicate the more nuanced takes in the topic, which seem to generally not be as inherently viral in their communication).
Above explains why it may work within the scope of theory despite being a poor method, but the success rate of methods like these is generally low enough to not be useful.
I'll give it more attention if they actually release conclusive benchmarks showing that it works instead of simply claiming it works, which is a big difference.
> The (Shannon) entropy of a variable X is defined as > H(X)=-sum_(x)P(x)log_2[P(x)]
> bits, where P(x) is the probability that X is in the state x, and Plog_2P is defined as 0 if P=0.
The X they input into that formula is a function that chooses one of the tokens according to the probability in that step. Isn't that a good definition of a random variable?
However, we can define it as a quantity with respect to different values. But the entropy of a variable as estimated by the model is generally not the actual entropy of the variable, and this gets worse for sequences -- we can maybe upper bound the entropy of a sequence when measuring it, but this is not always a useful or important quantity for us to have.
For more info, please see https://people.math.harvard.edu/~ctm/home/text/others/shanno...
But in this case, it means that the underlying point in embedding space doesn't map clearly to only one specific token. That's not too different from when you have an idea in your head but can't think of the word.
Humans are similar. A human's reality is built on the experiences (senses) it's been feed. There definitely are several major differences, the obvious one being that we have a different sensory input than an LLM, but there are others, like human's having a instinctual base model of reality, shaped by the effects of natural selection over our ancestors.
Just like an LLM can't tell if the reality it's been fed actually corresponds to the "truer" outside reality (you could feed an LLM lies like the sky is plaid in such a way that it would report that it's true), a human can't tell if the reality it's been fed actually corresponds to a "truer" outside reality (humans could be feed lies like we are in true reality, when we're actually all NPCs in a video game for a higher level).
The LLM can't tell if it's internal reality matches an outside reality, and humans can't tell if their internal reality matches an outside reality, because both only have the input they've received to go on, and can't tell if it's problematic or it's incomplete.
You're basically saying that no computer program can work, because if you randomly generate a computer program then most of them don't work.
What do you imagine a statistical output is ? and why do you imagine you can't be certain about it ? LLM are not picking words out of a bag at random and neither are they just blindly picking the most frequent words in the training set. What do you imagine all that computation is doing?
>given that it has no model of the meaning of any of the words in its output to compute certainty in the form of correspondence with reality?
Says who ? I mean basically all the research (quite a few) on the topic points to LLMs having a pretty good idea of the certainty and truth of their outputs internally. Some pretrained models even have the logit probabilities directly correspond to the probability of being right (https://imgur.com/a/3gYel9r).
Statistics is not magic. LLMs clearly have a model of the meaning of the words they use amongst many other things.
- sample multiple logits and branch (we maybe could with the old text completion API, but this no longer exists)
- add in a reasoning token on the fly
- stop execution, ask the user, etc.
But a visualization of logprobs in a query seems like it might be useful.
1- option top_logprobs allows you not just to get the most likely token, but the top most likely tokens.
You can branch, by just chosing any point in your generated string and feed it back to the LLM, for example: { "user":"what is the colour of love?", "assistant":"the colour of love is"}
It's true that it will add an "assistant" tag, wand old completions was better for this.
Are you saying the beginning of the article where it describes how the next token is predicted, how it’s possible to know the distribution of possible next tokens, isn’t accurate?
It also has no concept of what it means for the choice of token to be an “error” or not, or what a “correct” answer would be.
If I put a weight on one side of a die, and I roll it, the die is not more confident that it will land on that side than it would be otherwise, because dice do not have the ability to be confident. Asserting otherwise shows a fundamental misunderstanding of what a die is.
The same is true for LLMs.
But there are algorithms with stopping conditions (Newton-Raphson, gradient descent), and you could say that an answer is "uncertain" if it hasn't run long enough to come up with a good enough answer yet.
Yes, LLMs have a pretty good idea of the uncertainty and truth of their predictions internally. https://news.ycombinator.com/item?id=41418486
- An LLM knowing when it is lying is not the same thing as its internal state being able to “reveal the truthfulness of statements”. The LLM does not know when it is lying, because LLMs do not know things.
- It is incapable of lying, because lying requires possessing intent to lie. Stating untrue things is not the same as lying.
- As the paper states shortly afterwards, what it actually shows is “given a set of test sentences, of which half are true and half false, our trained classifier achieves an average of 71% to 83% accuracy”. That’s not the same thing as it being able to “reveal the truthfulness of statements”.
No intellectually honest person would claim that this finding means an LLM “knows when it is lying”.
You keep saying the same nonsense over and over again. A LLM does not know things so... What kind of argument is that ? You're working backwards from a conclusion that is nothing but your own erroneous convictions on what a "statistical model" is and are undertaking a whole lot of mental gymnastics to stay there.
There are a lot of papers there that all try to approach this in different ways. You should read them and try to make an honest argument and that doesn't involve "This doesn't count because - claim that is in no way empirically or theoretically validated."
I argued that LLMs do not have the capacity to have ideas or to know things, and you tried to prove me wrong by providing examples of papers that show, for example, that LLMs have internal states that can be used to predict the likelihood that what they will output will be facts. But that doesn’t disprove what I said, because that’s not what it means to have ideas or know things. By definition, only conscious beings can do those things.
If a machine is doing things previously before ascribed to "conscious beings" then it's on you to tell me why the machine is not conscious. Hopefully something other than the circular - "It cannot be conscious so it is not conscious".
But whatever. I hadn't quite realized this had devolved into a debate on consciousness. I think that's on me but I have no interest in a back and forth on such an ill-defined, ill-understood concept.
You don't know what consciousness is, what is required of it or what makes it tick in you, you have no way of proving one way or another anybody else has it. It's extremely silly then don't you think to make such bold declarations on what doesn't have it ? especially with circular arguments.
What difference does it make if you won't call it conscious if it does anything a conscious being does ? That's just semantics.
> It’s extremely silly then don’t you think to make such bold declarations on what doesn’t have it?
I don’t find it particularly bold to respond to your assertion that a piece of mathematics is sentient life by stating that you haven’t proven that it is, and that in the absence of that proof, the most rational position is to continue to believe that it is not, as we have done for millennia. The burden of proof is on you.
> if it does anything a conscious being does
You haven’t shown that it can do anything that only conscious beings can do.
Being able to generate a passable approximation of text that might follow some prompt doesn’t mean that it understands the prompt, or its answer. As an obvious example, if you give LLMs maths problems, they change their answers if you change the names of the people in the question. They’re not actually doing maths.
> Notice anything? It’s not just that the performance on MathGLM steadily declines as the problems gets bigger, with the discrepancy between it and a calculator steadily increasing, it’s that the LLM based system is generalizing by similarity, doing better on cases that are in or near the training set, never, ever getting to a complete, abstract, reliable representation of what multiplication is.[0]
[0] https://garymarcus.substack.com/p/math-is-hard-if-you-are-an...
That is your assertion. I'm not failing to understand anything. I'm simply telling you that you are stating an unproven assertion. This is why i don't like to debate consciousness.
Unless you believe in magic then the only thing that would stop whatever is running 'Newton-Ralph' from "knowing" roots if you are even right is that's it's not the kind of computation that "knows", not because it's a computation.
>I don’t find it particularly bold to respond to your assertion that a piece of mathematics is sentient life by stating that you haven’t proven that it is, and that in the absence of that proof, the most rational position is to continue to believe that it is not, as we have done for millennia. The burden of proof is on you.
The brain computes and unless you believe in a soul or something similar then that is all the brain does to produce consciousness. Computation is substrate independent[0]. Whether it is chemical reactions and nerve impulses or transistors in chips or even pulleys, it does not at all matter what is performing this computation.
Consciousness is clearly an emergent property. Your neurons are not conscious and they do not do conscious things and yet you believe you are conscious. "piece of mathematics" is entirely irrelevant here.
>You haven’t shown that it can do anything that only conscious beings can do. Being able to generate a passable approximation of text that might follow some prompt doesn’t mean that it understands the prompt, or its answer.
I know LLMs understand because of the kind of responses i get to the kind of queries i give them. This is how we probe and test understanding in humans.
>As an obvious example, if you give LLMs maths problems, they change their answers if you change the names of the people in the question.
No they don't. If you'd actually read that apple paper (i assume that's what's you are referring to), you would see that GPT-4o, o1-mini and o1-prievew do not shift above or below the margin of error numbers on 4/5 on the synthetic benchmarks they created. Definitely not for the ones that were just changing of names. So this is blatantly wrong. Changing names literally does nothing for today's state of the art LLMs
That Gary Marcus blog is idiotic but i don't expect much from gary marcus. There is not a single human on this planet that can perform arithmetic unaided (no calculator/writing down numbers) better than SOTA LLMs today. I guess humans don't understand or do math.
Not to mention that you can in fact train transformers that will generalize perfectly on addition.[1]
[0] https://www.edge.org/response-detail/27126
[1]https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mec...
Personally, until recently I can only recall people saying things along the lines of “applying the model indicates that we can state this fact about the data with this much confidence”, never “the model has this much confidence” in some truth statement, especially one independent of its training data.
It doesn't really bother me if they're mindless. It doesn't seem essential to me that we have free will, even
I've published extensive benchmarks: https://cleanlab.ai/blog/trustworthy-language-model/
You can instantly play with an interactive demo: https://tlm.cleanlab.ai/
I wonder if speculative decoding could help here? E.g. have some small model draft predictions for the branches and parallel and have to big model verify the most promising one.
Edited several times: I think to avoid this problem the answer of the LLM should be constrained in expression (say Yes or No, fill the blanks, etc). I think in that case we would have a decreasing sequence of the entropy for next token predictions.
I'm getting a little tired of people thinking I believe everything I read and publish. If you claim to have invented a time machine, a teleportation device, a phone to call the dead or if you take pictures back in time of course someone should document every tiny technical detail you've shared with the world. (preferably without repeatedly stating the obvious)
The idea a reader would believe everything strikes me as rather hilarious. Even if just a robot. LLMs should aid those skilled in the art who desire to make the same with the materials but it would be silly if it uncritically reproduced the description of your warp drive, your parallel universe detector, mr fusion, sentient black goo, channelings and remote viewings, alien encounters, bigfoot sightings, shape shifting lizard experiences, quantum computer or memristors.
The current stage of extracting the essense of reason from LLMs feels a lot like attempts to extract gold from iron in the medieval ages.
This is a subtle and understandable mistake, but I do suspect it's why they note at the top "A big caveat, there have been no large scale evals yet for Entropix, so it’s not clear how much this helps in practice. But it does seem to introduce some promising techniques and mental models for reasoning." I would like to see more evidence that High Entropy, Low Varentropy when deciding on a single token measurably corresponds with bad outcomes before accepting that there is any merit to this approach.
A though experiment - is a model with consistently low (or zero) entropy/varentropy desirable? First, it essentially means that the model makes no distinction in the semantics of different sequences of tokens in its answers, which due to the way models are trained also indicates that it probably makes no makes no distinction in the semantics of different sequences of tokens when processing input, which is bad, because that's not how language works. It also probably means that all the information encoded in the model's weights is "uncompressed" and doesn't generalize properly - the model may know that the sky was blue yesterday because it's in its training data, but how is it to know if it was blue today, or if it would be blue on a fictional planet with all the same physical characteristics as Earth? It's like saying you prefer your model to be overfit.
Another thought experiment - when you're starting a sentence, does it matter in the slightest whether you are highly predisposed to using "the" (low entropy+varentropy), split between about using "the" or "a" (low entropy, high varentropy), thinking about using many different definite/demonstrative words with no clear preference (high entropy, low varentropy), or thinking about using many different definite/demonstrative words with a clear preference to "the" (high entropy+varentropy)? It doesn't mean you're uncertain of the semantic meaning of the answer you're about to give. If you were to do as they suggest and take it as an indicator to think more deeply before responding, you'd not only waste time in your response (this is literally the same thing as when people say "um" and "uh" a lot when talking, which is considered bad) but distract yourself from the choice of answering with the right semantics with the choice of starting with the right word, which doesn't actually matter.
Speaking more abstractly or philosophically, why could a model never internalize something read between the lines? Humans do, and we're part of the same physical system — we're already our own kinds of computers that take away more from a text than what is explicitly there. It's possible.
return true;
There, I didn't need a paper to answer the question.