I think what you need is an architecture that emits a fully built answer directly from memory after working on it for a while, not one that builds it up token by token.
[0] https://www.hopsworks.ai/dictionary/rlhf-reinforcement-learn...
Like if I tell you: "I have a box, I place 3 balls in it and remove 1, how many balls are left". You have to assume that the box was empty before, that I want to know how many balls are left in the box, not on the side, that the box isn't leaking, etc. This is you making up facts that weren't present in the question.
It is, however, interesting that the “basic” word-spewing ability we now share with LLMs seems vital to the intelligence process.
I have no idea how that reckoning phase would be implemented.
The part where we can "analyse what we just said (or were about to say) (or what we just typed and didn't press send yet!) and conceptualise it as a whole to evaluate its veracity" involves going back-and-forth in yourself. A loop. LLMs are the part inside the loop, and so if you want to get better results, you have to feed the output back to LLM.
This is, arguably, part of what the "conversation mode" does, by always including the growing message log in each request - which is why e.g. GPT-4 is good at correcting a mistake once you say, without pointing it out, that there exist a mistake.
It is also good at adding a mistake, so this is completely unrelated. It just rolls to get another random.
What you should add is that ChatGPT does better when it writes out every step, so what you said previously is still true just that your example doesn't properly support it.
Edit: To expand on that, a humans inner thought would have caught the mistake at that point and corrected it by writing more instead of ending the text. The LLM basically never does that unless you tell it to fix the answer.
I don't think you can have functional intelligence without a willingness to fill in details you believe are missing, and that will not always work well. Consider how much effort we spend reinforcing in children how to recognize and suppress what is fantasy vs. reality, and to make them favour telling the truth. It shouldn't be surprising at all that we get "hallucinations" unless/until we put a massive effort into reinforcing that distinction with LLMs too.
Most of those people are doing it intentionally though - e.g. politicians when you ask them a question they don't want to answer.
> I don't think you can have functional intelligence without a willingness to fill in details you believe are missing
I disagree. A big part of intelligence is realising/admitting you don't know. That you are missing information. And you don't just fill it up with BS. You going find that information.
> Consider how much effort we spend reinforcing in children how to recognize and suppress what is fantasy vs. reality, and to make them favour telling the truth
Do we? I don't think children have problems telling the difference between fantasy and reality. If a child isn't telling the truth, most of the time it's not because they can't tell the difference between truth and falsehood, but because they are deliberately lying to you for one reason or another - e.g. they don't want to get punished.
That's distinct from the failure of communication. I believe GP meant something closer to what we'd call a "brain fart", where you read the text correctly, but "understood" something different.
> I don't think children have problems telling the difference between fantasy and reality. If a child isn't telling the truth, most of the time it's not because they can't tell the difference between truth and falsehood
Do you have children? I've been saying that GPT-3.5 and GPT-4 failure modes are disturbingly similar to the failure modes of my own kid, who happens to be 4.5 right now. I mean it.
It's a realization that hit me when I started noticing her "context window" growing - when she'd go into story mode, telling some fantasy stuff like having a sister giraffe who flies a helicopter or whatnot, I could tell that, if she doesn't mention something again within 30 seconds, it would never be mentioned again - a forgotten detail. That window of time kept growing over subsequent months, and is too large to notice now, but hey, the entire way she'd construct her stories was very similar to what you get when you prompt the LLM for a story and just let it keep writing.
Anyway, on the parenting stuff:
> it's not because they can't tell the difference between truth and falsehood, but because they are deliberately lying to you for one reason or another - e.g. they don't want to get punished.
That's true at a later age. Early on, even at 3 y.o., they get genuinely confused about reality.
That would be one example, yes.
But we also often interpret a question in the context of the person asking to mean something different than what the question actually says. A concrete example of a simple question that people will often outright hesitate to answer directly is "do you want a fair society?" Almost everyone can answer that just "yes" if they interpret the question as genuinely meaning just what it objectively said. But what people subjectively interpret that question to mean tends to vary based on assumptions about how the political views of the person asking match your own and their intention for asking.
Sometimes, yes, but also because people misunderstand are make invalid inferences.
> I disagree. A big part of intelligence is realising/admitting you don't know. That you are missing information. And you don't just fill it up with BS. You going find that information.
Notice how you inferred missing information here. Ironically you inferred the wrong intent behind what I said. I did not suggest "filling it up with BS" was useful nor desirable. It is a possible interpretation of what I wrote, I will accept, but it requires assumptions - one does not follow logically from the other.
What I actually suggested is that a willingness to fill in details is necessary, not that filling it in no matter what or without having something reasonable to fill in is necessary.
I agree that a big part of intelligence is realizing and admitting you don't know. I does not conflict with what I said. With the caveat that you can not always admit to not knowing every little detail you don't know - if you did you'd be unable to function.
My point was that we rarely "know" anything with 100% certainty, and so the question is not if we assume, but at what perceived level of certainty we set the bar. If I write "2+2=4", it is not a great sign of intelligence to ask me if I am using standard arithmetic or use special rules, unless you have context to suggest I might be or I have reason to interpret it as you being a smart-ass about it.
It is a core sign of intelligence to understand from context which bits of information it is likely justified to fill in when missing based on the fact that it makes a statement, or a behaviour, consistent if interpreted using that information.
So to restate, the willingness to fill in details is essential to functional intelligence, because without it we'd be forever unable to draw conclusions without continuing to seek clarifications of additional unstated assumptions.
Just reading this involves making assumptions about the degree to which my use of every individual words conforms to standard English usage, and indeed starts with the assumption that this is English to start with. Granted it is an assumption with exceedingly high certainty, but I have not told you I am, so it remains an inference whose certainty will get ever closer to 100% as the length of the conversation makes alternatives more and more unlikely.
When the uncertainty increases, sure, the intelligent choice at some level becomes to admitting that you don't know. Learning how to recognize the threshold is also essential to functional intelligence. Assuming much and too little both leads to conversations that go nowhere.
> Do we?
Yes, we do. We tell kids to not make things up in contexts where we expect them to answer us based on reality all the time. I'd be willing to claim that not an adult alive has not been told to stick to the truth a multitude of times. How effective that admonishment is certainly varies, and yes we sometimes also teach kids there are contexts where lies are better (whether we teach them that intentionally or not). Older children will certainly be better suited, including in terms of brain development that does not per se require reinforcement, to tell facts and fiction apart, but young children tend to hold a whole range of beliefs that appear genuine that has no relation to reality, and for many people some such beliefs persist for a long time when uncorrected.
Except that's exactly what LLMs do due to the way they work.
It's statistically guessing the next token. Unfortunately, just because statistically the next token is among the most likely to appear doesn't mean the sentence it forms is correct or even makes any sense.
Frankly, the whole way LLMs work is kind of absurd in the light of what we are trying to do.
https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
The darn algorithm actually has randomness built into it - and it's an absolutely necessary component.
Sufficient reinforcement that "I don't know" should be the most likely answer in some contexts is certainly important for intelligence.
It’s a language model. It models the statistical properties of human languages.
Here is a direct pair of quotes from a recent conversation with GPT4:
> Me: Answer in a single sentence, please. Do you, or do you not know the contents of Dr. Franz 1994 paper "Code Generation on the Fly: A key to portable software"?
> ChatGPT: No, I do not know the contents of Dr. Franz's 1994 paper "Code Generation on the Fly: A Key to Portable Software."
Is it prone to being evasive and waffling and not wanting to admit when it does not know, and preferring to jump to conclusions and try to get away with generalities, yes. Asking about this paper is one of my repeated test cases because it does so badly (there are few online sources on it, but his paper is online; other than that one of my blog posts and the Wikipedia article make up the bulk of text about it). And so it did when I tried it last - it waffled on about the (unrelated) concept of semantic encoding from NLP.
As far as I can tell, it is correct: It does not know the contents of the paper. It barely understands the high-level concepts involved (
> it doesn’t process the concept of “know” and “don’t know”, heck it doesn’t even process the concept of true and false.
> It’s a language model. It models the statistical properties of human languages.
It can explain them and use them. We don't know enough about what "knowing" something or "processing" a concept means in terms of human thought processes to know whether there's a meaningful distinction between the level at which an LLM processes these concepts vs. humans or whether there is a meaningful distinction between reasoning and intelligence vs. "modelling the statistical properties of human languages".
LLMs work by predicting the next bunch of words - it's advanced auto complete. If most of the training data replies "I don't" to the question 'Do you, or do you not know the contents of Dr. Franz 1994 paper "Code Generation on the Fly: A key to portable software"?' then that's what the LLM will say.
It's half useful for answering frequently asked questions but don't expect it to evaluate its current state and give you an accurate answer.
> We don't know enough about what "knowing" something or "processing" a concept means in terms of human thought processes to know whether there's a meaningful distinction between the level at which an LLM processes these concepts vs. humans or whether there is a meaningful distinction between reasoning and intelligence vs. "modelling the statistical properties of human languages".
But we do know what LLMs do, model language. Not knowledge. Not “thought”. Language.
And the way we get them to spit out output that’s satisfactory to us is just absurd. If you read the link I posted earlier, you will know that if you set the “temperature” of an LLM to zero, it just repeats itself talking in circles. It’s only by adding randomness to its “next token” search that we get output that possibly satisfactory.
Would you want a calculator that gets regularly gets addition wrong?
If you told me that thing about the box, I'd assume you meant a box that doesn't leak, not because of making that fact up in my brain, but because you're verbally describing a system to me, and it's implied you'll tell me the most relevant features of the system in order to transmit the idea of it from your mind to mine. Otherwise I'd be right to complain that you are maliciously hiding important information in your description.
Problem is, of course, we don't have a common preestablished language context with an LLM, at least not any more than what is the prevalent one in its training materials ("Ask" vs. "Guess" cultures, etc).
And that is pretty much exactly what an LLM does: It predicts the most likely item of data to fill in. That their training is as of yet insufficient to make them predict reasonable things in all situations should be far less surprising than how quickly we've gotten to a point where we can communicate well enough with them that these conversations are meaningful at all.
One of the fascinating things about these discussions to me is that they reveal an astonishing amount of assumptions about human thought processes that appear to be just marginally above LLM hallucinations in rigor.
I guess yeah, that's correct :)
But it's also not an artifact, but within the very definition of human interaction. We need to be able to plug holes. Otherwise conversation evolves into a list of guarding caveats after each and every message.
OTOH Twitter is a great example of how different cultures develop different assumptions about what is reasonable to fill the gaps in with. It's notorious for how people just take whatever wrong and unintended meaning they want from controversial tweets. Some times on purpose, granted, but lots of other times not really.
I concur it's a very interesting topic!
Absolutely.
So the big question is to what extent what LLMs do is this, but without (yet) having been sufficiently trained on what is reasonable to fill in unknown gaps with, vs. whether there is a bigger issue.
I'm pretty hopeful that it's down to more training on when "I don't know" should be the most likely output, but I think we really don't know until more work has been done on it...
Lots of separate phenomenon can cause our minds to have "wrong" or incomplete information, calling them all "hallucinations" is reductive and just serves to trivialize the vast differences in operation between human minds and LLMs.
Sure, in the exact same way LLM operates within context of everything it learned. It so happens that it's "cultural context" is derived from ours, by means of the training data, which is why it's so similar.
> If you give someone that question and then say "wrong, there was already a ball in the box!" they're not going to say "haha, silly me, I hallucinated"
And yet people are doing that to LLMs. "Haha, look at that LLM hallucinating", where half the time the prompt is misleading or wrong.
> they're going to stop hanging out with you.
That's the social equivalent of RLHF.
I mean... I ask it to give me Python code and it gives me code that doesn't work and makes up libraries that don't actually exist. That's not a prompt problem. A human would understand there's an implicit "code that actually works and isn't pulled out of your ass".
The LLM is operating similarly in a lot of cases. It doesn't have an IDE with intellisense acting as an immediate feedback mechanism.
A human certainly doesn't do that in my experience.
Apparently LLMs know when they are making stuff up (or so I keep getting told that over and over by others here).
But clearly they are terrible at communicating about when they are making things up which is a big problem to me.
I wish they would just say when they don't know something rather than make up BS. Plenty of humans are perfectly capable of acknowledging when they don't know something.
And ChatGPT will directly contradict the prompt even. I asked it yesterday what the value of a non-interest-bearing bond is and it replied that it's valuable because of the interest you collect from it just because that's the usual answer.
Unless we are actually doing symbolic reasoning like a theorem proof? But even then are we ever completely certain the proof is correct? This doesn't come naturally to us.
I have been to a grocery store at some point in my life. That is 100%. I trust you also have statements you can be certain of. Humans are 100% certain about many things.
While those are highly likely, and so it's reasonable to "round up" and claim to be 100% sure, the probability of it being true is most certainly not 100%.
We're talking about self reported probabilities humans are aware of. And yes, I am actually 100% sure of some facts about myself. (I have a sister, I live in an apartment). I am aware, abstractly, that such things may not be true in a sort of plato's cave shadows etc, but I will for all intents and purposes act and believe that they are certain. If such things were proven to not be true, I would be shocked to my very core; I am shocked because it broke a notion I had, if I was aware in my actual core of the possibility that it may not be true, I wouldn't be shocked.
To survive, our biological machinery must make assumptions and round probabilities. To do otherwise would be to be paralysed with indecision. (what if there is some small chance an elaborate deathtrap has been sprung that will kill me if I move even slightly at this very moment? what about the next moment?)
When your self-reported probability is out of whack with reality you will be prone to confidently making claims that are false and believe them.