Hacker News Reader

Comment by skybrian | Hacker News Reader

> it is very rare for GPT4 not to be able to understand why it went wrong.

That's not what's going on. GPT4 can't know the real reason why it wrote anything. It can never tell you the real reason why it made a mistake. Instead, we can say that GPT4 is fairly good at inventing plausible explanations about why someone did something, but it's no more accurate explaining its own mistakes than it would be at explaining one of your mistakes. [1]

A better explanation of what happened is that asking for an explanation resulted in chain-of-thought reasoning, and by telling it that its previous answer was wrong, it was biased to pick a different answer.

But this is often fragile. You'd want to do the same test multiple times, and compare regular chain-of-thought reasoning with what happens when you explicitly tell it that a particular answer is wrong.

[1] https://skybrian.substack.com/p/ai-chatbots-dont-know-why-th...

View on HN

> GPT4 can't know the real reason why it wrote anything. It can never tell you the real reason why it made a mistake.

Define "know" such that humans have it and GPT-4 does not have it.

Epistemology 101. Statistical pattern matching has no epistemological truth beyond a correspondence of linguistic statistics.

It is merely statistical probability, which hardly can be classified as an epistemological basis. If we pretend for a moment it is, the ontologies formed are most certainly peculiar, and we can expect ideologies that emerge out of the ontologies are problematic.

Beyesians accept an epistemological foundation of statistical priors updated by experience.

Predictive processing is well established in the neuroscience community.

Capital T Truth is really of very little interest to anyone outside of faith-based epistemology like religion.

The issue is that the definition of “statistics” is anchored in a magnitude of frequency of glyphs. The “information” is fabricated in this regard, pulled up out of the ether, and by decree christened as “meaning”.

Numbers carry no meaning, nor do the magnitudes arbitrarily assigned to meaning. The map is not the territory.

Have you heard of information theory?

Numbers can mean anything. A multitude of numbers as voltage potentials and ion gradients sufficiently describe your brain.

Biology manifests this arrangement as a brain, without which this arrangement would also be similarly meaningless.

Your argument against deriving meaning from statistics completely ignores that the brain also works this way.

> Have you heard of information theory?

> Your argument against deriving meaning from statistics completely ignores that the brain also works this way.

The brain is not predicting it's compressing.

The current prevailing theory in neuroscience is in fact the brain is a prediction engine.

https://en.m.wikipedia.org/wiki/Predictive_coding

> The brain is not predicting it's compressing

Compression requires prediction, therefore your brain requires prediction.

Some form of prediction being used by the higher-level neurons doesn't make the brain a prediction engine.

I'm not sure who claimed the brain was a "predictive engine" or what that means exactly. The OP specifically referenced predictive coding which describes precisely what is meant, and has empirical support.

If you meant this as a comparison to machine learning, then a predictive coding model closely matches.

> The “information” is fabricated in this regard, pulled up out of the ether, and by decree christened as “meaning”.

No, not fabricated, but inferred from a structured corpus of information generated by other semantic processes (humans).

> Numbers carry no meaning, nor do the magnitudes arbitrarily assigned to meaning.

Prove it.

> The map is not the territory.

Except if the territory is information, in which case the map is literally the territory. Knowledge is information, is it not?

Physicists are very interested in capital T Truth. They work with explanations not just statistical priors.

This talk by Feynman is slightly related:

https://youtu.be/obCjODeoLVw

Statical pattern matching is how human brains work.

> Epistemology 101. Statistical pattern matching has no epistemological truth beyond a correspondence of linguistic statistics. It is merely statistical

Your belief in epistemic truth is a statistical inference from your perception of apparently reliable causality. How do you ground this inductive inference?

Your attempt to appeal to epistemology 101 with a casual dismissal as "mere statistical probability" covers this deep, gaping maw. Bayesian inference reduces to classical logic when all probabilities are pinned to 0 and 1, but in what circumstances can we actually demonstrably infer absolute certainty? None that I can think of, except one's own existence.

I did not lay claim to epistemic truth. I referenced epistemology. That is, there are a plurality of epistemologies, all of which have their own epistemic truth mechanics.

So I would agree with you, entirely!

I was merely pointing out that statistical correlation alone affords no epistemological basis.

Okay, this isn't quite what you asked but it's close: we can decide to go on an errand and remember what we were doing even if we didn't write it down. (Though sometimes we do forget, so shopping lists are useful.)

This doesn't mean our justifications are accurate, but at least we do sometimes have "debug logs," some memories (perhaps faint and inaccurate) of internal thoughts that we can use to try to reconstruct what we were thinking.

An LLM cannot remember anything it didn't write down. It can't do that at all. They would need to change the API.

LLM doesn't have any other runtime state but for what it wrote down (assuming the user even provides it). For comparing with humans, I think LLMs are not like our minds, but like the inner voice. Chat history is the working memory.

Yep, and we have the text. It’s very transparent that way.

But it might be worth thinking about what a satisfying explanation of the LLM’s behavior would look like. It would have to be a high-level summary of the calculations that were done to predict each new token. Knowing which previous tokens got attention and what higher-level concepts it calculated would help. (Also, if the temperature isn’t zero, there’s a random number generator involved.)

The LLM doesn’t have access to that, once the calculation is done. It has the text, same as us. It can guess what might have happened, same as us. When it guesses it’s going to anthropomorphize by picking an explanation that would make sense for a human, because it’s trained to write justifications that a human would write.

> An LLM cannot remember anything it didn't write down. It can't do that at all.

Ok, but I'm not sure what that has to do with knowledge. You can ask ChatGPT to reflect on and correct its own answers, so it clearly has access to its own outputs, which is a form of memory.

By default ChatGPT keeps a context of 8000 tokens, which includes user provided prompts. That's the size of the working memory.

> It can never tell you the real reason why it made a mistake.

From my experience it can. Sometimes it writes the code and I have to point out that it produces a wrong result probably because of some function's behavior. GPT-4 can figure out the rest and fix it.

However, in general, it can go circles giving conflicting statements.

OP knows it looks like that but is trying to explain what might cause that instead of genuine understanding.

That's slightly different. It can sometimes spot a bug in some code it wrote, in the same way it could spot a bug in the code someone else wrote. It could also give a human-like justification for why the bug is there.

But that's different from a real explanation for why it wrote the code with a bug. A real explanation would have to be rather non-human.

>That's not what's going on. GPT4 can't know the real reason why it wrote anything.

Boy do I have news for you.

People can't recreate previous mental states, it's all plausible post-hoc rationalization.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/

And the rationalizations aren't always grounded either. The brain is just fine making up completely bogus explanations you believe to be true but couldn't possibly be so.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522/

https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf

The conscious self is at best a co-pilot. A Co-pilot that has shockingly little insight into the operations of the plane but thinks he knows much more.

Even your senses gets some pretty big post-processing by your brain you're consciously unaware of.

For example, ever wondered why the first second seems longer when you suddenly look at a clock ?

Human beings may not have the best visual acuity, but we are quite good at distinguishing colors and exceptional at processing visual information. A significant portion of our brain is dedicated to processing the visual stimuli from our binocular vision.

Nevertheless, binocular vision necessitates that both eyes focus on a single point. Our eyes can track a focused target as it moves, but they are unable to smoothly scan across our visual field. This is too difficult to process, so our eyes naturally jump from one point to another. Yet, during these jumps, known as "saccades," our brain does not even attempt to comprehend the blurry sensations that come in. In effect, you temporarily go completely blind!

However, you don't feel like you went blind, right? Your vision doesn't go black or anything, and it seems like you could see the entire time. This is because our brains fill in the lag while our eyes are moving, creating the illusion that we were seeing something instead of being blind during that time. Strangely enough, our brains fill it in with *what we see when they stop*! In other words, our brains go back and edit our memory of a split second ago to make it seem like we were looking at our new point of focus the whole time our eyes were moving to actually see it.

This phenomenon occurs when you look at the second hand of a clock. When you glance over at the clock, your brain tells you that you were looking at it for longer than you actually were, adding the time your eyes were moving but not yet seeing the clock. If the second hand was moving during this time, your brain will believe the second hand was in its new position longer than it actually was. Based on this flawed memory, it will appear as if the second hand stayed in that position for longer than a second.

Yes, of course. We often don't know our real motivations either, our justifications are often self-serving and made up, and more generally there's a lot we don't know about how the brain works.

But we do at least have some memories of our own thought processes and some experience observing our own reactions to things. You can get better at introspection. Attempting to understand and explain your own mistakes is useful.

An LLM won't be able to do this sort of learning without changing its training or architecture somehow.

Actually just added a long paragraph showing how even memories or real time sense data is pleasant fabrication. Also every time you remember something, you're not really recalling it. Your brain rewrites your memory every time you "remember" often introducing false data or inconsistencies. It's one of the reasons human memory is so fallible and inadmissible.

That said, I don't disagree with the overall point of what you've said. Attempted Introspection is useful. The point I'm making is to not dismiss LLM "introspection" simply because it's rationalization. It can be extremely useful too.

Fair enough!

The part I see missing for an LLM is learning from feedback about whether its justifications "work" for explaining its own actions. Keep in mind that an LLM is not human and its justifications often shouldn't look like human justifications. But it's going to be scored by how human its justifications look, so you will never get that. (With the exception of things explicitly trained in like knowing its own cut-off date - that's quite a non-human justification for not knowing something.)

Also, it's easy to come up with examples showing that we do remember what we were thinking, even if we didn't say it out loud. A simple example might be running an errand. Of course, sometimes we do forget, which is why a shopping list is useful.

An LLM cannot do that at all. It writes it down or it's lost. It can only pretend to remember an internal thought it had.

>An LLM cannot do that at all. It writes it down or it's lost.

I mean this is likely a solvable architecture issue at some point once we have the computing power to perform 'excess' computation and re-learn quickly. Consciousness, in my mind is a form of 'writing it down' and very rarely the primary thought itself, that is consciousness is reflection.

We see improvements in GPT behavior when allow externalized consciousness via reflection. It's able to 'see' its thoughts written, reflect on if they are correct, and then make corrections to improve the quality of its final answer.

The biggest hold up in LLMs is the amount of effort needed to retrain the LLM and ensure it's not gone off the rails. Currently they take months to train, now if Nvidia somehow makes the hardware a million more times powerful like they want to, will it take the training time down to minutes or hours?

>its own cut-off date - that's quite a non-human justification for not knowing something.)

That sounds very human to me... "I worked in the X industry from date Y to Z" is something people say all the time as to put a qualification date on the information they have about something.

Yes, there's no physical reason it couldn't be fixed. However, I think it's less a matter of computing power, and more one of architecture and training.

The inference-time code could be changed to keep a log of which neurons were activated the most when it generated a token. But then, how would we train it to use that log and report it accurately? Training on Internet documents isn't enough.

Chain-of-thought reasoning helps as a workaround. However, beware that even then, the justifications aren't a reliable explanation. There's an interesting paper on this:

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

https://arxiv.org/abs/2305.04388

> People can't recreate previous mental states, it's all plausible post-hoc rationalization. [...]

How could you possibly claim people are simply incapable of recreating mental states and that all they do is plausible post-hoc rationalization? Surely people can sometimes remember what they were thinking before making a decision? And in fact I would think people can do this more frequently than GPT can. Or are you really claiming those studies proved an implausible negative here?

"what they were thinking before making a decision?" Human brains are notoriously unreliable, we lie to ourselves all the time, we can't trust our memories.

> "what they were thinking before making a decision?" Human brains are notoriously unreliable, we lie to ourselves all the time, we can't trust our memories.

"Unreliable" is a far cry from "lie to ourselves all the time". Some of the time, sure. But all the time? Everything I'm telling myself is a post-hoc fabrication just like with GPT? Like you're telling me I can't even trust my memory of who my mom and dad are, or which country I live in? I'm justifying all of these post-facto somehow?

Perhaps those specific details are just a few heavily reinforced data points. Perhaps they become more and more statistically likely to be true the more they are reinforced?

Or perhaps they aren't, who knows. Regardless, is this a hypothesis that leads to anything useful?

The sense data you consciously perceive is pleasant fabrication. Moreover, every time you remember something, you're not really recalling it. Your brain rewrites your memory every time you "remember" (what you're really doing is remembering the last time you remembered it), often introducing false data or inconsistencies. It's one of the reasons human memory is so fallible and inadmissible.

The reason I linked the split brain experiments is to show that what you think is the reason for making a decision, even if you "remember" the thought processes before can just be wrong. Asking one side to perform an action and asking the other side to explain the reason for performing the action ended up producing some really wild results.

> The reason I linked the split brain experiments is to show that what you think is the reason for making a decision, even if you "remember" the thought processes before can just be wrong. Asking one side to perform an action and asking the other side to explain the reason for performing the action ended up producing some really wild results.

"Can" just be wrong and "produces some" wild results, sure. In the previous comment were claiming they're always fabrications (as, I note, seems to be the case with GPT!), not just that they could be.

The fact that both GPT and the human brain can sometimes fabricate things similarly does not mean the human brain (or GPT for that matter) does this all the time, let alone that they do this equally frequently, or always to equal degrees. The studies might be showing surprising results, but you're greatly exaggerating what the studies actually show.

Memories are always part-fabrications.

Our brain store "key" moments it thinks it is important, then make up the rest of it. But this process is not really trustworthy on the finer details and is extremely receptive to any sort of suggestion because it simply doesn't have any factual data, so happily jumps to anything which seems to fit the narrative. This is especially true in scenarios where you experience strong emotions. Anything feels like to fit (like, someone paints a scenario which creates the same type of emotion) our brain just accepts it as a matter of fact and puts the puzzle in.

It's why implanting false memories is extremely easy and leading questions work so well. https://en.m.wikipedia.org/wiki/Lost_in_the_mall_technique

Anything that you remember from your life is mostly fabrications with a couple of actual memories used as a scaffolding, and your brain just built the rest of the structure around it - constantly changing it.

And yet we have a very hard time believing or accepting this reality, preferring to believe in our own infallibility.

I think some of what is misleading you here is something you think the brain does well, which it does not...

The brain is not a 'good' memory device. It is a really terrible one. Pick up any hard drive and write data to it and it's going to be a massively more faithful and reliable memory device.

Memory is not the power of the brain. Filtering is by far one of our greatest brain powers. At extremely low power levels we can input absolutely massive amounts of sensing data and capture information form noise. For example have someone put up a wall of pictures in a room. When you first see it your brain will capture a bunch of information, but every time after that it turns into more filtered noise you don't capture. And this noise is really easy to switch out with different objects and thoughts. When one of your trickster friends switches out the picture of your grandmother for Hitler, it's exceptionally unlikely you'll notice.

But what's even better about this is when someone points this out to you, how they point it out to you will influence "when you noticed it". If they say something like "We switched out this picture months ago and you noticed it pretty quickly" your brain will rewrite history when you recall this gag later and you will think you noticed it months beforehand unless you go to a pretty significant effort not to be gaslit.

Exactly, the people making confident statements about the inner workings of the brain are just as ridiculous as the people making the same statements about LLMs.

We should have a healthy dose of humility here and realize we know very little about either the brain or the meaning hidden in those ~200 billion parameters.

First: these studies are unrelated to AI and mostly predate modern LLMs, so no, they draw no direct comparisons. At best, this is pointing out two surprisingly similar behaviors we observe in LLMs and the actual functioning of the human mind.

Second: Being unable to reconstruct a prior mental state =/= remembering a thought process. I'll use an accursed programing analogy because we are on an accursed programing website: it's like the difference between debug logs and time-travel debugging. In plain language: humans retroactively reason about their thought process based on the artifacts produced by that thought process. The quality and reliability of these artifacts will vary heavily, but the reasoning itself is always backward-looking and speculative in nature.

That's all fine, but if the claim is really "I cannot know which exact neurons were firing at some moment in the past" (that's what time-travel debugging would be), that's not a particularly insightful or useful statement. I'm pretty sure I can reconstruct some states of my brain for some moments in the past sufficiently adequately to be able to explain my reasoning in that moment without fabricating anything. The claim was that this is always impossible for all humans, which seems quite obviously false, and in any case, unsupported by the studies linked.

Honestly the proof of evidence that you can recall your memory state needs to be proven by you, as I don't believe any one has brought forth evidence of what you're suggesting.

Almost everything I've seen and read about this suggests the opposite. Activating memory changes memory. Intermediate events between the initial activation and the final questioning can also change those memories. And furthermore, that you are only aware of your conscious memories (and even most of that is forgotten) and that memory is accessed unconsciously all the time.

It's all post ad hoc.

Humans work in a completely different way, and human performance is irrelevant to the obvious, systemic limitations of LLMs.

You haven't addressed the substance of the comment you're responding to, only diverted attention. Unless you're claiming that humans are nothing but LLMs...

> Humans work in a completely different way

Citation?

Hugo Mercier was recently on Sean Carroll's podcast talking about the concepts from his new book The Enigma of Reason. The gist of it is that most of the function of what we call reasoning doesn't exist to implement logical reasoning, but rather evolved to provide the social utility of providing "reasons" to convince or placate other humans. I guess what I'm saying is we may be closer to ChatGPT than we think - we may not usually be logically reasoning when we provide reasons, but inventing after the fact reasons that are statistically likely to be accepted for what we did or what we want to do.

Podcast: https://www.preposterousuniverse.com/podcast/2023/04/17/233-...

Reply on news.ycombinator.com