Language models can explain neurons in language models | Hacker News Reader

Language models can explain neurons in language models

openai.com

688 pointsmfiguiere477 comments

Of note:

"... our technique works poorly for larger models, possibly because later layers are harder to explain."

And even for GPT-2, which is what they used for the paper:

"... the vast majority of our explanations score poorly ..."

Which is to say, we still have no clue as to what's going on inside GPT-4 or even GPT-3, which I think is the question many want an answer to. This may be the first step towards that, but as they also note, the technique is already very computationally intensive, and the focus on individual neurons as a function of input means that they can't "reverse engineer" larger structures composed of multiple neurons nor a neuron that has multiple roles; I would expect the former in particular to be much more common in larger models, which is perhaps why they're harder to analyze in this manner.

Funny that we never quite understood how intelligence worked and yet it appears that we're pretty damn close to recreating it - still without knowing how it works.

I wonder how often this happens in the universe...

The battery (Voltaic Pile, 1800) and the telegraph (1830s-1840s) were both invented before the electron was discovered (1897).

Also Darwin published a theory of evolution, and Mendel discovered genetics, before anyone even thought of the term "double helix".

No need for genetics to understand evolution...

But they "didn't know how it worked", DNA was not discovered yet

Hell, consider all the technology that was developed before science really got going.

No need to know about electrons to understand electricity.

Imitation -> emulation -> duplication -> revolution is a very common pattern in nature, society, and business. Aka “fake it til you make it”.

Think of business / artistic / cultural leaders nurturing protégés despite not totally understanding why they’re successful.

Of course those protégés have agency and drive, so maybe not a perfect analogy. But I’m going to stand by the point intuitively even if a better example escapes me.

> that we're pretty damn close to recreating it

Is that evident already or are we fitting the definition of intelligence without being aware?

If you spent any time with GPT-4 it should be evident.

If you spent even more time with GPT-4 it would be evident that it is definitely not. Especially if you try to use it as some kind of autonomous agent.

Still a while to go. I think there's at least a couple of algorithmic changes needed before we move to a system that says "You have the world's best god-like AI and you're asking me for poems. Stop wasting my time because we've got work to do. Here's what I want YOU to do."

Its vast limitations in anything reasoning-based are indeed evident.

While impressive GPT-4 isn't intelligent or close to intelligence. It's an impressive statistical model but doesn't have the ability to reason

Compare gpt-4 with a baby and you'll see that predicting the next word in sequence is not human intelligence

I watched a magician's show where he made his assistant hover in mid-air. It is evident that the magician has mastered levitation.

Looks more like a Chinese Room to me.

Evolution created intelligence without even being intelligent itself

How do you know that’s true?

Evolution is just a description of a process, it isn't a tangible thing.

Because we eat and breath through the same tube

Because evolution is random change + natural selection

You're proposing that statistics is intelligent?

Evolution is not a single orchestrator. It is merely the natural result of a simple mechanical process over a timescale that exceeds the lifetime of the common human.

Yep, we don't know all constituents of buttermilk, nor how bread stales (there's too much going on inside). But it doesn't prevent us to judge their usefulness.

I feel like OAI's approach is kind of wrong. GPT4 is still just text transformation/completion with multi-headed attention for better prediction of the next word that should follow (versus only looking at the previous word).

In human brains, language is only a way to communicate thoughts in concept form, though we also seem to use language to communicate abstract thoughts to ourselves to break them apart/down in a way (imo).

I'd love to see someone train a model on the level of GPT4 to generate abstract thoughts/ideas based on input/context and then pair this model with GPT4 co-operatively and continue to train, such that the flow of abstract ideas is parsed by GPT. But like...how do you even train a model that operates on abstract ideas, there doesn't seem to be any way to do this.

Starting a fire is easy to do even if you don't know how it works.

Appropriate analogy in more ways than one. Particularly given the unpredictable consequences of doing so.

We are probably creating something that looks like our intelligence but it works in a different way.

An example: we are not very good at creating flight, the one birds do and humans always regarded as flight, and yet we fly across half the globe in one day.

Going up three meters and landing on a branch is a different matter.

Not that weird if you think about it, our intelligence simultaneously measly and amazing as it is, was the product of trial, error, and sheer dumb luck. We could think of ourselves as monkeys with typewriters, eventually we'll get it right.

No matter what it is probably easier to inspect a language model while it works, than the language module of a human while he speaks.

We don’t know…

I like the idea. Note that LLMs have some skill at decoding sequential dense vectors in the human brain

https://pub.towardsai.net/ais-mind-reading-revolution-how-gp...

so why not have them decode sequential dense vectors of their own activations?

As for the majority scoring poorly, they suggest that most neurons won't have clear activation semantics so that is intrinsic to the task and you'd have to move to "decoding the semantics of neurons that fire as a group"

I don't think this is showing LLMs performing decoding. They're just using the LLM to propose possible words. The decoding is done by using another model to score how well a proposed word matches brain activity, and using that score to select a most likely sequence given the proposals from the LLM.

We know that complex arrangements of neurons are triggered based on input and generating output that appears to have some intelligence to many humans.

The more interesting question is why are intelligence/beauty/consciousness emergent properties that exist in our minds.

There is no evidence that intelligence runs on neurons. Yes, there are neurons in brains, but there's also lots of other stuff in there too. And there are creatures that exhibit intelligent properties even though they have hardly any neurons at all. (An individual ant has only something like 250000 neurons, and yet they're the only creatures beside humans that managed to create a civilization.)

This is not a good take. Yes there is a lot more going on in brains than just neuronal activity, we don’t understand most of it. But understanding neurons and their connections is necessary (but not sufficient) to understanding what we consider intelligence. Also, 250k is a lot of neurons! Individual ants, as well as fruit flies which have even fewer neurons, show behavior we may consider intelligent. Source: I am not a scientist, but I work in neuroscience research

What's the argument that understanding neurons is necessary?

Perhaps intelligence is like a black box input to our bodies (call it the "soul", even though this isn't testable and therefore not a hypothesis). The mind therefore wouldn't play any more of a role in intelligence than the eye. And I'm not sure people would say the eye is necessary for understanding intelligence.

Now, I'm not really in a position to argue for such a thing, even if I believe it, but I'm curious what argument you might have against it.

> There is no evidence that intelligence runs on neurons.

1. Neurons connect all our senses and all our muscles.

2. Neurons are the definitive difference between the brain and the rest of the body. There is “other stuff” in the brain, but it’s not so different from the “other stuff” that’s in your rear end.

Don’t underestimate what a neuron can do. A single artificial neuron can fit a logistic regression model. A quarter of a million is on then scale of some our our largest AI, and biological neurons are far more connected than ANN. An ant quite likely has a more powerful brain than GPT-4.

> 2. Neurons are the definitive difference between the brain and the rest of the body. There is “other stuff” in the brain, but it’s not so different from the “other stuff” that’s in your rear end.

Our digestive systems appear to be important to our behaviour, though. Some recent work in mice showed that if colonised with bacteria from faeces of humans with autism, the mice would begin to show autistic behaviours.

So, not sure your argument here is especially strong.

Maybe the neurons are the hardware layer. The software is represented by the electronic activity. There is a good video https://youtu.be/XheAMrS8Q1c about this topic.

If you really want to present ants as a civilization, I don't think a single ant is a meaningful unit of that civilization comparable to a single human. A colony, perhaps - but then that's a lot more neurons, just distributed.

A single human will die permanently, alone, miserable, and unproductive. The comparison is apt.

What else would intelligence run on?

Every cell in our body, and every bacterium living in a body (e.g. gut flora), contribute to our intelligence. It looks plausible (to me) that there's one "top cell" among them that represents the "person", others just contributing via layered signals, but whether this "top cell" is a neuron or another kind of cell is unknown.

Ah, so you're not studied in neuroscience with some revolutionary theory. A shame.

In what way a civilization?

I'll repost a comment via Reddit that I think makes this case [0]:

Ants have developed architecture, with plumbing, ventilation, nurseries for rearing the young, and paved thoroughfares. Ants practice agriculture, including animal husbandry. Ants have social stratification that differs from but is comparable to that of human cultures, with division of labor into worker, soldier, and other specialties that do not have a clear human analogy.

Ants enslave other ants. Ants interactively teach other ants, something few other animals do, among them humans. Ants have built "supercolonies" dwarfing any human city, stretching over 5,000 km in one place. And ants too have a complex culture of sorts, including rich languages based on pheromones.

Despite the radically different nature of our two civilizations, it is undeniable from an objective standpoint that this level of society has been achieved by ants.

[0]: https://www.reddit.com/r/unpopularopinion/comments/t2h1vs/an...

Nature created humans to understand nature. We created GPT4 to understand ourselves.

Humans are the universe asking who made it.

Yes, or put a bit more elegantly, 'The cosmos is within us. We are made of star-stuff. We are a way for the universe to know itself.' — Carl Sagan

Yes, "all serve "gpt4!" Get a grip, pretty please.

A mirror of ourselves*

That's beautiful until you think about it.

Humans so far have done a great job at destroying nature faster than any other kind could.

And GPT4 was created for profit.

There is no evidence that any of those are emergent properties. It’s no more or less logical than asserting they were placed there by a creator.

There is evidence that people believe that GTP-4 is intelligent since it can solve things like the SATs. But if you start taking away weights one by one at some point those same people will say it isn't intelligent. A NN with 3 weights cannot solve any problems that humans believe requires intelligence. So where did it come from? I don't know, but it clearly emerged as the NN got bigger.

It's possible that the comment you're responding to is referring to the new paper out of Stanford/Google on emergence as a 'mirage'. The gist of it is that the shape of the curve for 'metric' vs 'params' is dependent upon the type of metric. For classification it has a certain shape, for accuracy it has another, etc. The authors claim the observation that these curves are determined by the choice of metric means that it isn't true emergence, but rather is due to the way in which GPTs are typically trained to predict the next token (and then beam search or typical sampling is often applied afterwards, etc).

The paper is somewhat new so I haven't done a proper review to know if it's solid work yet, but this may offer some context for some of the comments in this thread.

I suspect that there's a sweet spot that combines a collection of several "neurons" and a human-readable explanation given a certain kind of prompt. However, this "three-body problem" will probably need some serious analytical capability to understand at scale

here's a clue, start your research

Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our approach on a wide range of benchmarks for natural language understanding. Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied. For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on question answering (RACE), and 1.5% on textual entailment (MultiNLI).

My favorite line: “Our overall procedure is quite compute intensive.”

Should that be measured in number of nuclear power plants needed to run the computation? Or like, fractions of a small star’s output?

Yes but they are from openai so they can just write papers that say whatever they want to say without minding the metrics and then pretend like it is some kind of science.

> Which is to say, we still have no clue as to what's going on inside GPT-4 or even GPT-3, which I think is the question many want an answer to.

Exactly. Especially:

> ...the technique is already very computationally intensive, and the focus on individual neurons as a function of input means that they can't "reverse engineer" larger structures composed of multiple neurons nor a neuron that has multiple roles;

This paper just brings us no closer to explainability in black box neural networks and is just another excuse piece by OpenAI to try to please the explainability situation that has been missing for decades in neural networks.

It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently.

> It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently.

Like say, in court to detect if someone is lying? Or at an airport to detect drugs?

You don't even have to look that far ahead. Apparently, people are already using ChatGPT to compile custom diet plans for themselves, and they expect it to take into account the information they supply regarding their allergies etc.

But, yes, those are also good examples of what we shouldn't be doing, but are going to do anyway.

>Apparently, people are already using ChatGPT to compile custom diet plans for themselves, and they expect it to take into account the information they supply regarding their allergies etc.

Evolution is still doing it's thing.

Those cases sound like Darwin Awards mediated by high technology

Is it really fair to say this brings us “no closer” to explainability?

This seems like a novel approach to try to tackle the scale of the problem. Just because the earliest results aren’t great doesn’t mean it’s not a fruitful path to travel.

> the explainability situation that has been missing for decades in neural networks.

Is this true? I thought explainability for things like DNNs for vision made pretty good progress in the last decade.

> It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently.

Doesn't this criticism also apply to people to some extent? We don't know what the purpose of individual brain neurons is.

People are better understood intuitively. We understand how people fail and why. We can build trust with people with some degree of success. But machine models are new and can fail in unpredictable ways. They also get deployed to billions of users in a way that humans do not, and deployed in applications that humans do not. So its certainly useful to try to explain neural networks in as great of detail as we can.

Or we can build trust using black box methods like we do with humans, e.g., extrapolating from past behavior, administering tests, and the like.

As a person I can at least tell you what I do and don't understand about something, ask questions to improve/correct my understanding, and truthfully explain my perspective and reasoning.

The machine model is not only a black box, but one incapable of understanding anything about its input, "thought process", or output. It will blindly spit out a response based on its training data and weights, without knowing the difference whether it true or false, meaningful or complete gibberish.

As a person, you can tell what you think you do and don't understand, and you can explain what you think your reasoning is. In practice, people get both wrong all the time. People aren't always truthful about it, either, and there's no reliable way to tell if they are.

I built a toy neural network that runs in the browser[1] to model 2D functions with the goal of doing something similar to this research (in a much more limited manner, ofc). Since the input space is so much more limited than language models or similar, it's possible to examine the outputs for each neuron for all possible inputs, and in a continuous manner.

In some cases, you can clearly see neurons that specialize to different areas of the function being modeled, like this one: https://i.ameo.link/b0p.png

This OpenAI research seems to be feeding lots of varied input text into the models they're examining and keeping track of the activations of different neurons along the way. Another method I remember seeing used in the past involves using an optimizer to generate inputs that maximally activate particular neurons in vision models[2].

I'm sure that's much more difficult or even impossible for transformers which operate on sequences of tokens/embeddings rather than single static input vectors, but maybe there's a way to generate input embeddings and then use some method to convert them back into tokens.

[1] https://nn.ameo.dev/

[2] https://www.tensorflow.org/tutorials/generative/deepdream

This tool is really lovely, great work!

I'd be curious to see Softmax Linear Units [1] integrated into the possible activation functions since they seem to improve interpretability.

PS: I share your curiosity with respect to things like deep dream. My brief summary of this paper is that you can use GPT4 to summarize what's similar about a set of highlighted words in context which is clever but doesn't fundamentally inform much that we didn't already know about how these models work. I wonder if there's some diffusion based approach that could be used to diffuse from noise in the residual stream towards a maximized activation at a particular point.

[1] https://transformer-circuits.pub/2022/solu/index.html

this is wonderful, thanks for sharing!

"This work is part of the third pillar of our approach to alignment research: we want to automate the alignment research work itself. A promising aspect of this approach is that it scales with the pace of AI development. As future models become increasingly intelligent and helpful as assistants, we will find better explanations."

On first look this is genius but it seems pretty tautological in a way. How do we know if the explainer is good?... Kinda leads to thinking about who watches the watchers...

> How do we know if the explainer is good?

The paper explains this in detail, but here is a summary: an explanation is good if you can recover actual neuron behavior from the explanation. They ask GPT-4 to guess neuron activation given an explanation and an input (the paper includes the full prompt used). And then they calculate correlation of actual neuron activation and simulated neuron activation.

They discuss two issues with this methodology. First, explanations are ultimately for humans, so using GPT-4 to simulate humans, while necessary in practice, may cause divergence. They guard against this by asking humans whether they agree with the explanation, and showing that humans agree more with an explanation that scores high in correlation.

Second, correlation is an imperfect measure of how faithfully neuron behavior is reproduced. To guard against this, they run the neural network with activation of the neuron replaced with simulated activation, and show that the neural network output is closer (measured in Jensen-Shannon divergence) if correlation is higher.

> The paper explains this in detail, but here is a summary: an explanation is good if you can recover actual neuron behavior from the explanation.

To be clear, this is only neuron activation strength for text inputs. We aren't doing any mechanistic modeling of whether our explanation of what the neuron does predicts any role the neuron might play within the internals of the network, despite most neurons likely having a role that can only be succinctly summarized in relation to the rest of the network.

It seems very easy to end up with explanations that correlate well with a neuron, but do not actually meaningfully explain what the neuron is doing.

Eh, that's why the second check I mentioned is there... To see what the neuron is doing in relation to the rest of the network.

Why is this genius? It's just the NN equivalent of making a new programming language and getting it to the point where its compiler can be written in itself.

The reliability question is of course the main issue. If you don't know how the system works, you can't assign a trust value to anything it comes up with, even if it seems like what it comes up with makes sense.

I love the epistemology related discussions AI inevitably surfaces. How can we know anything that isn't empirically evident and all that.

It seems NN output could be trusted in scenarios where a test exists. For example: "ChatGPT design a house using [APP] and make sure the compiled plans comply with structural/electrical/design/etc codes for area [X]".

But how is any information that isn't testable trusted? I'm open to the idea ChatGPT is as credible as experts in the dismal sciences given that information cannot be proven or falsified and legitimacy is assigned by stringing together words that "makes sense".

> But how is any information that isn't testable trusted? I'm open to the idea ChatGPT is as credible as experts in the dismal sciences given that information cannot be proven or falsified and legitimacy is assigned by stringing together words that "makes sense".

I understand that around the 1980s-ish, the dream was that people could express knowledge in something like Prolog, including the test-case, which can then be deterministically evaluated. This does really work, but surprisingly many things cannot be represented in terms of “facts” which really limits its applicability.

I didn’t opt for Prolog electives in school (I did Haskell instead) so I honestly don’t know why so many “things” are unrepresentable as “facts”.

I bet GPT is really good at prolog, that would be interesting to explore.

"Answer this question in the form of a testable prolog program"

Seems relevant: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...

There is a longer-term problem of trusting the explainer system, but in the near-term that isn't really a concern.

The bigger value here in the near-term is _explicability_ rather than alignment per-se. Potentially having good explicability might provide insights into the design and architecture of LLMs in general, and that in-turn may enable better design of alignment-schemes.

I can almost hear the Animatrix voiceover: "At first, AI was useful. Then, we decided to automate oversight... The rest is history."

It also lags one iteration behind. Which is a problem because a misaligned model might lie to you, spoiling all future research with this method

It doesn't have to lag, though. You could ask gpt-2 to explain gpt-2. The weights are just input data. The reason this wasn't done on gpt-3 or gpt-4 is just because a) they're much bigger, and b) they're deeper, so the roles of individual neurons are more attenuated.

I had similar thoughts about the general concept of using AI to automate AI Safety.

I really like their approach and I think it’s valuable. And in this particular case, they do have a way to score the explainer model. And I think it could be very valuable for various AI Safety issues.

However, I don’t yet see how it can help with the potentially biggest danger where a super intelligent AGI is created that is not aligned with humans. The newly created AGI might be 10x more intelligent than the explainer model. To such an extent that the explainer model is not capable of understanding any tactics deployed by the super intelligent AGI. The same way ants are most probably not capable of explaining the tactics delloyed by humans, even if we gave them a 100 years to figure it out.

Safest thing to do, stop inverting and building more powerful and potentially dangerous systems which we can’t understand?

You're correct to have a suspicion here. Hypothetically the explainer could omit a neuron or give a wrong explanation for the role of a neuron. Imagine you're trying to understand a neural network, and you spend enormous amount of time generating hypotheses and validating them. Well the explainer might give you 90% correct hypotheses, it means you have 10 times less work to produce hypotheses. So if you have a solid way of testing an explanation, even if the explainer is evil, it's still useful.

It produces examples that can be evaluated.

https://openaipublic.blob.core.windows.net/neuron-explainer/...

Using 'im feeling lucky' from the neuron viewer is a really cool way to explore different neurons. And then being able to navigate up and down through the net to related neurons.

Fun to look at activations and then search for the source on the net.

"Suddenly, DM-sliding seems positively whimsical"

https://openaipublic.blob.core.windows.net/neuron-explainer/...

https://www.thecut.com/2016/01/19th-century-men-were-awful-a...

How do we know WE are good explainers :-)

Gpt2 answers to gpt3. Gpt3 answers to gpt4. Gpt4 answers to God.

> We are open-sourcing our datasets and visualization tools for GPT-4-written explanations of all 307,200 neurons in GPT-2, as well as code for explanation and scoring using publicly available models on the OpenAI API. We hope the research community will develop new techniques for generating higher-scoring explanations and better tools for exploring GPT-2 using explanations.

Aww, that's so nice of them to let the community do the work they can use for free. I might even forget that most of OpenAI is closed source.

Surely they are willing to shell a out a few bucks

LLMs are quickly going to be able to start explaining their own thought processes better than any human can explain their own. I wonder how many new words we will come up with to describe concepts (or "node-activating clusters of meaning") that the AI finds salient that we don't yet have a singular word for. Or, for that matter, how many of those concepts we will find meaningful at all. What will this teach us about ourselves?

First of all, our own explanations about ourselves and our behaviour are mostly lies, fabrications, hallucinations, faulty re-memorization, post hoc reasoning:

"In one well-known experiment, a split-brain patient’s left hemisphere was shown a picture of a chicken claw and his right hemisphere was shown a picture of a snow scene. The patient was asked to point to a card that was associated with the picture he just saw. With his left hand (controlled by his right hemisphere) he selected a shovel, which matched the snow scene. With his right hand (controlled by his left hemisphere) he selected a chicken, which matched the chicken claw. Next, the experimenter asked the patient why he selected each item. One would expect the speaking left hemisphere to explain why it chose the chicken but not why it chose the shovel, since the left hemisphere did not have access to information about the snow scene. Instead, the patient’s speaking left hemisphere replied, “Oh, that’s simple. The chicken claw goes with the chicken and you need a shovel to clean out the chicken shed”" [1]. Also [2] has an interesting hypothesis on split-brains: not two agents, but two streams of perception.

[1] 2014, "Divergent hemispheric reasoning strategies: reducing uncertainty versus resolving inconsistency", https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522

[2] 2017, "The Split-Brain phenomenon revisited: A single conscious agent with split perception", https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf

I'm not understanding the connection between your paragraphs here even after reading the first article.

Even if you accept classic theory (e.g. hemispheric localization and the homunculus) which most experts don't all this suggests is that the brain tries to make sense of the information it has and in sparse environments it fills in.

How does this make our behavior "mostly lies, fabrications, hallucinations, faulty re-memorization, post hoc reasoning" as most humans don't have a severed corpus callosum.

The discussion starts with:

"In a healthy human brain, these divergent hemispheric tendencies complement each other and create a balanced and flexible reasoning system. Working in unison, the left and right hemispheres can create inferences that have explanatory power and both internal and external consistency."

I think the point is that, in a non-healthy brain, the brain can create a balanced and flexible reasoning system that creates inferences that have explanatory power, but which may not match external reality. Oliver Sacks has a long bibliography of the weird things that can go on in brains.

But the bottom line is that introspection is not necessarily reliable.

He didn't say behavior. He said explanations of behaviour. Split brain experiments aside, this is pretty evident from other research. We can't recreate previous mental states, we just do a pretty good job (usually) of rationalizing decisions after the fact. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/

I'm reading this as our explanations for our own behavior as in why am I typing on this keyboard right now, in which case it's not evident at all.

The existence of cognitive dissonance suggested in your citation is in no way analogous to "our own explanations about ourselves and our behaviour are mostly lies, fabrications, hallucinations, faulty re-memorization, post hoc reasoning" and in fact supports the opposite.

Not supported by neuroimaging. Promoted without evidence or sufficient causal inference.

https://www.health.harvard.edu/blog/right-brainleft-brain-ri... :

> But, the evidence discounting the left/right brain concept is accumulating. According to a 2013 study from the University of Utah, brain scans demonstrate that activity is similar on both sides of the brain regardless of one's personality.

> They looked at the brain scans of more than 1,000 young people between the ages of 7 and 29 and divided different areas of the brain into 7,000 regions to determine whether one side of the brain was more active or connected than the other side. No evidence of "sidedness" was found. The authors concluded that the notion of some people being more left-brained or right-brained is more a figure of speech than an anatomically accurate description.

Here's wikipedia on the topic: "Lateralization of brain function" https://en.wikipedia.org/wiki/Lateralization_of_brain_functi...

Furthermore, "Neuropsychoanalysis" https://en.wikipedia.org/wiki/Neuropsychoanalysis

Neuropsychology: https://en.wikipedia.org/wiki/Neuropsychology

Personality psychology > ~Biophysiological: https://en.wikipedia.org/wiki/Personality_psychology

MBTI > Criticism: https://en.wikipedia.org/wiki/Myers%E2%80%93Briggs_Type_Indi...

Connectome: https://en.wikipedia.org/wiki/Connectome

You are talking about the popular narrative of “left brain” thinking being more logical and “right brain” thinking being more creative. You are correct this is unsupported.

The post you are replying to is talking about the small subset of individuals who have had their corpus callosum surgically severed, which makes it much more difficult for the brain to send messages between hemispheres. These patients exhibit “split brain” behavior that is well studied by experiments and can shed light into human consciousness and rationality.

This is not relevant to GP's comment. It has nothing to do with "are there fixed 'themes' that are operated in each hemisphere." It has to do with more generally, does the brain know what the brain is doing. The answer so far does not seem to be "yes."

Says who? There is actual evidence to support that our brain doesn't "know" what it is doing on a subconscious level? As far as I'm aware it's more that conscious humans don't understand how our brain works.

I think the correct statement is "so far the answer is we don't know"

Your response doesn't seem to be directly related to the previous poster's split-brain comments, but rather the popular misuse of the lateralization idea.

Agree I'm not a neuro subspecialist but I've listened to some talks at conferences out of interest and I don't think anyone still believes in this anymore. Anecdotally the few fMRI's I reported as a trainee didn't support this either.

Phantoms in the Brain is a fascinating book that deals with exactly this topic.

that is absolutely fascinating and also makes me extremely uncomfortable

Neurology is full of very uncomfortable facts. Here's one for you: there are patients who believe their arm is gone even though it's still there. When the doctor asks whose arm that is, they reply it must be someone else's. The brain can simply refuse to know something, and will adopt whatever delusions and contortions are necessary. Which of course leads to the realization that there could be things we're all incapable of knowing. There could be things right in front of our faces we simply refuse to perceive and we'd never know it.

Famously, our nose is literally right in front of our faces and the brain simply "post-processes" it out of the view.

After breaking my arm, split in two, pinching the nerve and making me unable to move it for about a year, I still feel as if the arm is "someone else's", as if I am moving an object in VR, not something which is "me" or "mine".

Oliver Sacks' A Leg To Stand On is a lengthy discussion of that, including his own experiences after breaking a leg---IIRC, at one point after surgery but before he starts physical therapy, he wakes up convinced that a medical student has played a prank by removing his leg and attaching one from a cadaver, or at least sticking a cadaver's leg under his blanket. (ISTR he tries to throw it out of bed and ends up on the floor.)

It's probably way worse than we can imagine.

Reading/listening to someone like Robert Sapolsky [1] makes me laugh I could have ever hallucinated about such a muddy, not even wrong concept as "free will".

Furthermore, between the brain and, say, the liver there is only a difference of speed/data integrity inasmuch as one cares to look for information processing as basal cognition: neurons firing in the brain, voltage-gated ion channels and gap junctions controlling bioelectrical gradients in the liver, and almost everywhere in the body. Why does only the brain has a "feels like" sensation? The liver may have one as well, but the brain being an autarchic dictator perhaps suppresses the feeling of the liver, it certainly abstracts away the thousands of highly specialized decisions the liver takes each second solving adequately the complex problem space of blood processing. Perhaps Thomas Nagel shouldn't have asked "What Is It Like to Be a Bat?" [2] but what is it like to be a liver.

[1] "Robert Sapolsky: Justice and morality in the absence of free will", https://www.youtube.com/watch?v=nhvAAvwS-UA

[2] https://en.wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat%3F

The biggest problem with the current popular idea of “free will” is that people think it means they’re ineffably unpredictable. They’re uncomfortable with the notion that if you were to simulate their brain in sufficient detail, you could predict thoughts and reaction. They take refuge in pseudoscientific mumbling about the links to the Quantum, for they have heard it is special and unpredictable.

And that’s just the polar opposite of having a meaningful will at all. It is good that you are pretty much deterministic. You shouldn’t be deciding meaningful things randomly. If you made 20 copies of yourself and asked them to support or oppose some essential and important political question (about human rights, or war, or what-have-you) they should all come down on the same side. What kind of a Will would that be that chose randomly?

This is why I suggest that curious individuals try a hallucinogen at least once*. It really makes the fragility of our perception, and how it’s held up mostly by itself, very apparent.

* in a safe setting with support, of course.

I can confirm lucid dreaming does the same.

Same. We are so, so profoundly not what it feels like we are, to most of us anyway.

I am morbidly curious how people are going to creatively explain away the more challenging insights AI gives us in to what consciousness is.

If that makes you uncomfortable you definitely should not go reading the evidence supporting the notion that conscious free will is an illusion.

https://www.mpg.de/research/unconscious-decisions-in-the-bra...

My impression is that the understanding of that research that comes up with statements like "But when it comes to decisions we tend to assume they are made by our conscious mind. This is questioned by our current findings" is based on dualistic reasoning.

The idea that there should not be any neural activity before a conscious decision is straight-up dualism---the intangible soul makes a decision and neural activity follows it to carry out the decision.

An alternative way of understanding that result is that the neural activity that precedes the "conscious decision" is the brain's mechanism of coming up with that decision. The "conscious mind" is the result of neural activity, right?

i always thought that the concept of free will didnt bother me. but it turns out i just didn’t understand what it implied. oh dear.

if it turns out that true, it’s truly amazing how well we convince ourselves that we’re in control.

but if our brain controls our actions and not our consciousness, then what is the purpose of consciousness?

Just wait until you notice how much humans do this day to day

"LLMs are quickly going to be able to start explaining their own thought processes better than any human can explain their own."

There is no "their" and there is no "thought process" . There is something that produces text that appears to humans like there is something like thought going on (cf the Eliza Effect), but we must be wary of this anthropomorphising language.

There is no self reflection, but if you ask an LLM program how "it" knows something it will produce some text.

> There is no self reflection, but if you ask an LLM program how "it" knows something it will produce some text.

To be clear, you're saying that we should just dismiss out-of-hand any possibility that an LM AI might actually be able to explain its reasoning step-by-step?

I find it kind of charming actually how so many humans are just so darn sure that they have their own special kind of cognition that could never be replicated. Not even with 175,000,000,000 calculations for every word generated.

That's a strawman since I didn't argue anything about humans being special. I don't think there is anything necessarily inherently special about human intelligence, I'm just advocating for caution around the language we use to talk about current systems.

All this talk of AGI and sentience and so on is premature and totally unfounded . It's pure sci fi, for now at least.

> I didn't argue anything about humans being special

Above you said about AI LMs:

> There is no "their" and there is no "thought process"

So, unless you're claiming that humans lack a thought process as well, then you're arguing that humans are special.

> All this talk of AGI and sentience and so on is premature and totally unfounded

I don't see any mention of AGI or sentience in this thread?

Also, I don't think anyone could read this transcript with GPT-4 and still claim that it's incapable of a significant degree of self-reflection and metacognition:

https://news.ycombinator.com/item?id=35880148

As we don't know for sure what is happening 100% within a neural network, we can say we don't believe that they're thinking and we would still need to define the word thinking. Once LLM's can self-modify, the word "thinking" will be more accurate than it is today.

And when Hinton says at MIT, "I find it very hard to believe that they don't have semantics when they consult problems like you know how I paint the rooms how I get all the rooms in my house to be painted white in two years time," I believe he's commenting on the ability of LLM's to think on some level.

In this case, I think we do if you will check out the paper (https://openaipublic.blob.core.windows.net/neuron-explainer/...). Their method is to

1. Show GPT-4 a GPT-produced text with the activation level of a specific neuron at the time it was producing that part of the text highlighted. They then ask GPT-4 for an explanation of what the neuron is doing.

Text: "...mathematics is done _properly_, it...if it's done _right_. (Take ..."

GPT produces "words and phrases related to performing actions correctly or properly".

2. Based on the explanation, get GPT to guess how strong the neuron activates on a new text.

"Assuming that the neuron activates on words and phrases related to performing actions correctly or properly. GPT-4 guesses how strongly the neuron responds at each token: '...Boot. When done _correctly_, "Secure...'"

3. Compare those predictions to the actual activations of the neuron on the text to generate a score.

So there is no introspection going on.

They say, "We applied our method to all MLP neurons in GPT-2 XL [out of 1.5B?]. We found over 1,000 neurons with explanations that scored at least 0.8, meaning that according to GPT-4 they account for most of the neuron's top-activating behavior." But they also mention, "However, we found that both GPT-4-based and human contractor explanations still score poorly in absolute terms. When looking at neurons, we also found the typical neuron appeared quite polysemantic."

Very true. In my opinion, in case there is a way to extract "Semantic Clouds of Words", i.e given a particular topic, navigate semantic clouds word by word, find some close neighbours of that word, jump to a neighbour of that word and so on, then LLMs might not seem that big of a deal.

I think LLMs are "Semantic Clouds of Words" + grammar and syntax generator. Someone could just discard the grammar and syntax generator, just use the semantic cloud and create the grammar and syntax by himself.

For example, in writing a legal document, a slightly educated person on the subject, could just use the relevant words put into an empty paper, fill in the blanks of syntax and grammar, alongside with the human reasoning which is far superior than any machine reasoning, till today at least.

The process of editing the GPT* generated documents to fix reasoning is not a negligible task anyway. Sam Altman mentioned that: "the machine has some kind of reasoning", not a human reasoning ability by any means.

My point is, that LLMs are two programs fused into one, "word clouds" and "syntax and grammar", sprinkled with some kind of poor reasoning. Their word clouding ability, is so unbelievable stronger than any human it fills me with awe every time i use it. Everything else is, just whatever!

I think they're much more than that. Or rather, if they're "Semantic Cloud of Words", they're still a hundred thousand dimensional clouds of words, and in those hundred thousand dimensions, any relationship you can think of, no matter how obscure, ends up being reflected as proximity along some subset of dimensions.

Looking at it this way, I honestly wouldn't be surprised if that's exactly how "System 1" (to borrow a term from Kahneman) in our brains works.

What I'm saying is:

> In my opinion, in case there is a way to extract "Semantic Clouds of Words", i.e given a particular topic, navigate semantic clouds word by word, find some close neighbours of that word, jump to a neighbour of that word and so on, then LLMs might not seem that big of a deal.

It may be much more of a deal than we'd naively think - it seems to me that a lot of what we'd consider "thinking" and "reasoning" can be effectively implemented as proximity search in a high-dimensional enough vector space. In that case, such extracted "Semantic Cloud of Words" may turn out to represent the very structure of reasoning as humans do it - structure implicitly encoded in all the text that was used as training data for the LLMs.

>if they're "Semantic Cloud of Words", they're still a hundred thousand dimensional clouds of words, and in those hundred thousand dimensions, any relationship you can think of, no matter how obscure, ends up being reflected as proximity along some subset of dimensions.

Yes, exactly that. That's what GPT4 is doing, over billions of parameters, and many layers stacked on top of one another.

Let me give you one more tangible example. Suppose Stable Diffusion had two steps of generating images with humans in it. One step, is taking as input an SVG file, with some simple lines which describe the human anatomy, with body position, joints, dots as eyes etc. Something very simple xkcd style. From then on, it generates the full human which corresponds to exactly the input SVG.

Instead of SD being a single model, it could be multimodal, and it should work a lot better in that respect. Every image generator suffers from that problem, human anatomy is very difficult to get right.[1] The same way GPT4 could function as well. Being multimodal instead of a single model, with the two steps discreet from one another.

So, in some use cases, we could generate some semantic clouds, and generate syntax and grammar as a second step. And if we don't care that much about perfect syntax and grammar, we feed it to GPT2, which is much cheaper to run, and much faster. When i used the paid service of GPT3, back in 2020, the Ada model, was the worst one, but it was the cheapest and fastest. And it was fast. I mean instantaneous.

>the very structure of reasoning as humans do it

I don't agree that the machine reasons even close to a human as of today. It will get better of course over time. However in some not so frequent cases, it comes close. Some times, it seems like it, but only superficially i would argue. Upon closer inspection the machine spits out non sense.

[1] Human anatomy, is very difficult to get right, like an artist. Many/all of the artists, point out the fact, that A.I. art doesn't have soul in the pictures. I share the same sentiment.

The text output of a llm is the thought process. In this context the main difference between humans and llms, is that llms can’t have internalized thoughts. There are of course other differences to, like the fact that humans have a wider gamut of input: visuals, sound, input from other bodily functions. And the fact that we have live training.

It's not clear whether or not LLMs have internal thoughts -- each token generation could absolutely have a ton of thought-like modelling in the hidden layers of the network.

What is known is that these internal thoughts get erased each time a new token is generated. That is, it's starting from scratch from the contents of the text each time it generates a word. But you could postulate that similar prompt text leads to similar "thoughts" and/or navigation of the concept web, and therefore the thoughts are continuous in a sense.

True, LLMs definetly has something that is "thought-like".

But todays networks lacks the recursion(feedback where the output can go directly to the input) that is needed for the type of internalized thoughts that humans have. I guess this is one thing you are pointing at by mentioning the continuousnes of the internals of LLMs.

Or maybe the human thought process isn't as sophisticated as we imagined.

I'm not arguing for or against that. It's more the implications of sentience and selfhood implicit in the language many use around LLMs.

This is it. This comprehension of the chats is symptom of something like linguistic pareidolia. It's an enforced face that is composed of some probabilistic incidents and wistfulness.

Pride comes before the fall, and the AI comes before humility.

Precisely.

When someone states definitively what LLMs can or cannot do, that is when you know to immediately disregard them as the waffling of uninformed laymen lacking the necessary knowledge foundations (cognitive/neuroscience/philosophy) to even appreciate the uncertainty and finer points under discussion (all the open questions regarding human cognition etc).

They don't know what they don't know and make unfounded assertions as result.

Many would do to refrain from speaking so surely about matters they know nothing about, but that is the internet for you.

https://en.wikipedia.org/wiki/Predictive_coding

Clearly there is some process going on to decide what word to pick. Why can’t we call that thinking?

What if you ask it to emit the reflexive output, then feed that reflexive output back into the LLM for the conscious answer?

What if you ask it to synthesize multiple internal streams of thought, for an ensemble of interior monologues, then have all those argue with each other using logic and then present a high level answer from that panoply of answers?

What if you do? LLMs don't have reflexive output or internal streams of thought, they are simply (complex) processes that produce streams of tokens based on an inputted stream of tokens. They don't have a special response to tokens that indicate higher-level thinking to humans.

LLMs seem to me to be the "internal streams of thought". I.e. it's not LLMs that are missing an internal process that humans have, but rather it's humans that have an entire process of conscious thinking built on top of something akin to LLM.

If you direct the model output to itself and don't view it otherwise, how is it not an "internal stream of thought"?

You seem to have high confidence in how LLMs work.

If the Gödel incompleteness theorem applies here, then the explanations are likely … incomplete or self-referential.

The Goedel Incompleteness Theorem has no straightforward application to this question.

It would if the language model did reasoning according rules of logic. But they don't. They use Markov chains.

To me it makes no sense to say that a LLM could explain its own reasoning if it does no (logical) reasoning at all. It might be able to explain how the neural network calculates its results. But there are no logical reasoning steps in there that could be explained, are there?

Honest question: are we sure that it doesn’t do logical reasoning?

IANAE but although an LLM meets the definition of a Markov Chain as I understand it (current state in, probabilities of next states out), the big black box that spits out the probabilities could be doing anything.

Is it fundamentally impossible for reasoning to be an emergent property of an LLM, in a similar way to a brain? They can certainly do a good impression of logical reasoning- better than some humans in some cases?

Just because an LLM can be described as a Markov Chain doesn’t mean it _uses_ Markov Chains? An LLM is very different to the normal examples of Markov Chains I’m familiar with.

Or am I missing something?

In any case, coemu is an interesting related idea to constrain AIs to thinking in ways we can understand better:

https://futureoflife.org/podcast/connor-leahy-on-agi-and-cog...

https://www.alignmentforum.org/posts/ngEvKav9w57XrGQnb/cogni...

What leads you to suspect that Gödel incompleteness may be relevant here?

There's no formal axiom system being dealt with here, afaict?

Do you just generally mean "there may be some kind of self-reference, which may lead to some kind of liar-paradox-related issues"?

The relevance is because all (all known buildable aka algorithmic, and sufficiently powerful) models of computation are equivalent in terms of formal computability, so if you could violate/bypass the Godel or Turing theorems in neural networks, then you could do it in a Turing machine, and vice versa. (That's my understanding, feel free to correct me if I'm mistaken)

Well... , yeah, but, these models already produce errors for other unrelated reasons, and like...

Well, what exactly would we be showing that these models can’t do? Quines exist, so there’s no general principle preventing reflection in general. We can certainly write poems (etc.) which describe their own composition. A computer can store specifications (and circuit diagrams, chip designs, etc.) for all its parts, and interactively describe how they all work.

If we are just saying “ML models can’t solve the halting problem”, then ok, duh. If we want to say “they don’t prove their own consistency” then also duh, they aren’t formal systems in a sense where “are they consistent (as a formal system)?” even makes sense as a question.

I don’t see a reason why either Gödel or Turing’s results would be any obstacle for some mechanism modeling/describing how it works. They do pose limits on how well they can describe “what they will do” in a sense of like, “what will it ‘eventually’ do, on any arbitrary topic”. But as for something describing how it itself works, there appears to be no issue.

If the task to give it was something like “is there any input which you could be given which would result in an output such that P(input,output)” for arbitrary P, then yeah I would expect such diagonalization problems to pop-up.

But a system having a kind of introspection about how it works, rather than answering arbitrary questions about its final outputs (such as, program output, or whether a statement has a proof), seems totally fine.

Side note: One funny thing: (aiui) it is theoretically possible for a oracle that can have random behavior, to act (in a certain sense) as a halting-oracle for Turing machines with access to the same oracle.

That’s not to say that we can irl construct such a thing, as we can’t even make a halting oracle for normal Turing machines. But, if you add in some random behavior for the oracles, you can kinda evade the problems that come from the diagonalization.

I commented in another answer but you can consult https://etc.cuit.columbia.edu/news/basics-language-modeling-...

Some training forms include entailment : “if A then B”. I hope this is first order logic which does have an axiom system :)

So is the word "word" but that seems to have worked out OK so far. I can explain the meaning of "meaning" and that seems to work OK too. Being self-referential sounds a lot more like a feature than a bug. Given that the neurons in our own heads are connected to each other and not any ground truth, I think LLMs should do just fine.

As long as lazy evaluation exists, self-reference is fine, no?

Hofstadter talks about something similar in his books.

That's probably one of the reasons why you'd use GPT-4 to explain GPT-2.

Of course, if you were trying to use GPT-4 to explain GPT-4 then I think the Gödel incompleteness theorem would be more relevant, and even then I'm not so sure.

Are there any examples of an LLM developing concepts that do not exist or cannot be inferred from its training set?

"Cannot be inferred from its training set" is a pretty difficult hurdle. Human beings can infer patterns that aren't there, and we typically call those hallucinations or even psychoses. On the other hand, some unconfirmed, novel patterns that humans infer actually represent groundbreaking discoveries, like for example much of the work of Ramanujan.

In a real sense, all of the future discoveries of mathematics already exist in the "training set" of our present understanding, we just haven't thought it all the way through yet. If we discover something new, can we say that the concept didn't exist, or that it "couldn't be inferred" from previous work?

I think the same would apply to LLMs and their understanding of the way we encode information using language. Given their radically different approach to understanding the same medium, they are well poised to both confirm many things we understand intuitively as well as expose the shortcomings of our human-centric model of understanding.

I'm really curious what kind of concept you might have in mind. Can you give any example of a concept that if an LLM developed that concept then it would meet your criteria? It might sound like a sarcastic question but it's hard to agree on the meanings of "concepts that do not exist" or "concepts that cannot be inferred" maybe you can give some examples.

EDIT: I see below you gave some examples, like invention of language before it existed, and new theorems in math that presumably would be of interest to mathematicians. Those ones are fair enough in my opinion. The AI isn't quite good enough for those ones I think, but I also think newer versions trained with only more CPU/GPU and more parameters and more data could be 'AI scientists' that will make these kinds of concepts.

It is by definition impossible for an LLM to develop a concept that 'cannot be inferred from its training set'.

On the other hand, that is an incredibly high bar.

Tautologically, every concept that anything (LLM, or human, or alien) develops can be inferred from the input data(e.g. training set), because it was.

No, it wasn't, language itself didn't even exist at one point. It wasn't inferred from training data into existence because such examples existed before. Now we have a dictionary of tens of thousands of words, which describe high level ideas, abstractions, and concepts that someone, somewhere along the line had to invent.

And I'm not talking about imitation nor am I interested in semantic games, I'm talking about raw inventiveness. Not a stochastic parrot looping through a large corpus of information and a table of weights on word pairings.

Has AI ever managed to learn something humans didn't already know? It's got all the physics text books in its data set. Can it make novel inferences from that? How about in math?

> No, it wasn't, language itself didn't even exist at one point.

Language took dozens of millennia to form, and animals have long had vocalizations. Seems like a natural building on top of existing features.

> Has AI ever managed to learn something humans didn't already know?

AlphaZero invented all new categories of strategy for games like Go, when previously we thought almost all possible tactics had been discovered. AIs are finding new kinds of proteins we never thought about, which will blow up the fields of medicine and disease in a few years once the first trials are completed.

Coming up with new language is rarely ever coming up with new concepts that didn't exist until the word. We come up with high-level abstractions because there already exists a material system to be described and modeled. Language that doesn't describe anything that already exists is more like babbling.

The training sets are so poorly curated we will never know...

I’m sure LLMs are quickly going to learn to hallucinate (or let’s use the proper word for what they’re doing: confabulate) plausible-sounding but nonsense explanations of their thought processes at least as well as humans.

I don't think we invent new words just for AI to explain its thought process to us better. AI may explain more elaborately in our language instead.

And if the LLM is the explainer, it can lie to us if 'needed'.

Based on my skimming the paper, am I correct in understanding that they came up with an elaborate collection of prompts that embed the text generated by GPT-2 as well as a representation of GPT-2's internal state? Then, in effect, they simply asked GPT-4, "What do you think about all this?"

If so, they're acting on a gigantic assumption that GPT-4 actually correctly encodes a reasonable model of the body of knowledge that went into the development of LLMs.

Help me out. Am I missing something here?

>If so, they're acting on a gigantic assumption

Yes the initial hypothesis that GPT-4 would know was a gigantic assumption. But a falsifiable one which we can easily generate reproducible tests for.

The idea that simulated neurons could learn anything useful at all was once a gigantic assumption too.

Neural networks aren’t simulated neurons…

Why does it have to understand how the LLMs are built? They have used gpt-4 to just build a classifier for each neuron's activition, and given the nlp abilities of gpt-4, the hope is that it can describe the nature of activation of the neurons.

After GPT-4 generates the hypothesis for a neuron they test it by comparing GPT-4's expectation for where the neuron should fire against where it actually fires.

If you squint it's train/test separation.

To me the value here is not that GPT4 has some special insight into explaining the behavior of GPT2 neurons (they say it's comparable to "human contractors" - but human performance on this task is also quite poor). The value is that you can just run this on every neuron if you're willing to spend the compute, and having a very fuzzy, flawed map of every neuron in a model is still pretty useful as a research tool.

But I would be very cautious about drawing conclusions from any individual neuron explanation generated in this way - even if it looks plausible by visual inspection of a few attention maps.

They also mention they got a score above 0.8 for 1000 neurons out of GPT2 (which has 1.5B (?)).

1.5B parameters, only 300k neurons. The number of connections is roughly quadratic with the number of neurons.

I thought they had only applied the technique to 307,200 neurons. 1,000 / 307,200 = 0.33% is still low, but considering that not all neurons would be useful since they are initialized randomly, it's not too bad.

This isnt exactly building an understanding of LLMs from first principles... IMO we should broadly be following the (imperfect) example set forth by neuroscientists attempting to explain fMRI scans and assigning functionality to various subregions in the brain. It is circular and "unsafe" from an alignment perspective to use a complex model to understand the internals of a simpler model; in order to understand GPT4 then we need GPT5? These approaches are interesting, but we should primarily focus on building our understanding of these models from building blocks that we already understand.

I've been working in systems neuroscience for a few years (something of a combination lab tech/student, so full disclosure, not an actual expert).

Based on my experience with model organisms (flies & rats, primarily), it is actually pretty amazing how analogous the techniques and goals used in this sort of research are to those we use in systems neuroscience. At a very basic level, the primary task of correlating neuron activation to a given behavior is exactly the same. However, ML researchers benefit from data being trivial to generate and entire brains being analyzable in one shot as a result, whereas in animal research elucidating the role of neurons in a single circuit costs millions of dollars and many researcher-years.

The similarities between the two are so clear that I noticed that in its Microscope tool [1], OpenAI even refers to the models they are studying as "model organisms", an anthropomorphization which I find very apt. Another article I saw a while back on HN which I thought was very cool was [2], which describes the task of identifying the role of a neuron responsible for a particular token of output. This one is especially analogous because it operates on such a small scale, much closer to what systems neuroscientists studying model organisms do.

[1] https://openai.com/research/microscope [2] https://clementneo.com/posts/2023/02/11/we-found-an-neuron

Yup, this article on predictive coding for example is particularly interesting.

Lots of parallels to how our brains are thought to work.

https://en.m.wikipedia.org/wiki/Predictive_coding

I don't follow. Neuroscience imaging tools like fMRI are only used because it is impossible to measure the activations of each neuron in a brain in real time (unlike an artificial neural network). This research paper's attempt to understand the role of individual neurons or neuron clusters within a complete network gets much closer to "first principles" than fMRI.

Right so it should be much easier w/ access to every neuron and activation. But the general approach is an experimental one where you try to use your existing knowledge about physics and biology to discern what is activating different structures (and neurons) in the brain. I agree w/ the approach of trying to assign some functionality to individual 'neurons', but I don't think that using GPT4 to do so is the most appealing way to go about that, considering GPT4 is the structure we are interested in decoding in the first place.

All of this seems to lead to something like this paper https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...

On the other hand, I find it plausible that it's fundamentally impossible to assign some functionality to individual 'neurons' due to the following argument:

1. Let's assume that for a system calculating a specific function, there is a NN configuration (weights) so that at some fully connected NN layer there is a well-defined functionality for specific individual neurons - #1 represents A, #2 represents B, #3 represents C etc.

2. The exact same system outcome can be represented with infinitely many other weight combinations which effectively result in a linear transformation (i.e. every possible linear transformation) of the data vector at this layer, e.g. where #1 represents 0.1A + 0.3B + 0.6C, #2 represents 0.5B+0.5C, and #3 represents 0.4B+0.6C - in which case the functionality A (or B, or C) is not represented by any individual neurons;

3. When the system is trained, it's simply not likely that we just happen to get the best-case configuration where the theoretically separable functionality is actually separated among individual 'neurons'.

Biological minds do get this separation because each connection has a metabolic cost; but the way we train our models (both older perceptron-like layers, and modern transfomer/attention ones) do allow linking everything to everything, so the natural outcome is that functionality simply does not get cleanly split out in individual 'neurons' and each 'neuron' tends to represent some mix of multiple functionalities.

Your last idea, that these models’ neurons are all connected in some way, makes me somewhat sceptical of this research by OpenAI. And that their technique of analysis may need to be more fractal or expansive to include groups of neurons, moving all the way up to the entire model.

> in order to understand GPT4 then we need GPT5?

I also found this amusing. But you are loosely correct, AFAIK. GPT-4 cannot reliably explain itself in any context: say the total number of possible distinct states of GPT-4 is N; then the total number of possible distinct states of GPT-4 PLUS any context in which GPT-4 is active must be at least N + 1. So there are at least two distinct states in this scenario that GPT-4 can encounter that will necessarily appear indistinguishable to GPT-4. It doesn't matter how big the network is; it'll still encounter this limit.

And it's actually much worse than that limit because a network that's actually useful for anything has to be trained on things besides predicting itself. Notably, this is GPT-4 trying to predict GPT-2 and struggling:

> We found over 1,000 neurons with explanations that scored at least 0.8, meaning that according to GPT-4 they account for most of the neuron’s top-activating behavior. Most of these well-explained neurons are not very interesting. However, we also found many interesting neurons that GPT-4 didn't understand. We hope as explanations improve we may be able to rapidly uncover interesting qualitative understanding of model computations.

1,000 neurons out of 307,200--and even for the highest-scoring neurons, these are still partial explanations.

There's little reason to think that predicting GPT-4 would be more difficult, only that it would be far more computationally expensive (given the higher number of neurons and much higher computational cost of every test).

Seems like OpenAI is grasping at straws trying to make GPT "go meta".

Reminds me of this Sam Altman quote from 2019:

"We have made a soft promise to investors that once we build this sort-of generally intelligent system, basically we will ask it to figure out a way to generate an investment return."

https://youtu.be/TzcJlKg2Rc0?t=1886

I have a similar feeling, they’ve potentially built the most amazing but commercially useless thing in history.

I don’t mean it’s not useful entirely, but I mean. It’s not useful in that it’s not deterministic enough to be trustworthy, it’s dangerous and really hard to scale therefore it’s more of an academic project than something that will make Altman as famous as Sergey Brin.

I personally take people like Hinton seriously too and think people playing with these things need more oversight themselves.

Time will tell. Anecdotally, I know several professional who find ChatGPT3.5 & 4 to be valuable and willing to pay for access. I certainly save more than $20 per month for my work by using ChatGPT to accelerate my day to day activities.

That’s why I was careful to say it’s not useless.

I wonder will someone please check the neurons associated to the petertodd and other anomalous glitch tokens (https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petert...)? I can see the github and I see that for any given neuron you can see associated tokens but I don't see how to do an inverse search.

Those were discovered by finding strings that OpenAI’s tokenizer didn’t properly split up. Because of this, they are treated as singular tokens, and since these don’t occur frequently in the training data, you get what are effectively random outputs when using them.

The author definitely tries to up the mysticism knob to 11 though, and the post itself is so long, you can hardly finish it before seeing this obvious critique made in the comments.

The ironic thing about lesswrong is that it’s quite the opposite in some fantastically oblivious ways.

Yeah, it’s quite strange indeed. Clearly people with decent educations but zero background in applied research/peer review. More concerned with the sound of their own voice than with whether or not their findings are actually useful (or even true).

Perhaps they are all on stimulants!

Thank you for your opinion on the post that I linked! I'm still curious about the associated neurons though.

Fair enough. You would need to use an open model or work at OpenAI. I assume this work could be used on the llama models - although I’m not aware of anyone has found these glitchy phrases for those models yet.

> You would need to use an open model or work at OpenAI.

The point of this post that we are commenting under is that they made this association public, at least in the neuron->token direction. I was thinking some hacker (like on hacker news) might be able to make something that can reverse it to the token->neuron direction using the public data so we could see the petertodd associated neurons. https://openaipublic.blob.core.windows.net/neuron-explainer/...

I think this method will do a poor job at explaining petertodd. These neuron explanations still have to fit within human language for this method to work, and the best you can do in the confines of human language to describe petertodd is to write a long article (just like that link) explaining the many oddities of it.

Would be interesting to try, though. I think it's likely that, due to the way glitch tokens happen, petertodd is probably an input neuron that is very randomly connected to a bunch of different hidden neurons. So it introduces some bizzare noise into a bunch of areas of the network. It's possible that some of these neurons are explainable on their own, but not within the broader context of petertodd.

ok but i still want to see it

For people overwhelmed by all the AI science speak, just spend a few minutes with bing or phind and it will explain everything surprisingly well.

Imagine telling someone in the middle of 2020, that in three years a computer will be able to speak, reason and explain everything as if it was a human, absolutely incredible!

In 2020 we were headed for an AI winter, according to all the hot takes:

https://www.bbc.com/news/technology-51064369

https://link.springer.com/article/10.1007/s13347-020-00396-6

https://blog.re-work.co/ai-experts-discuss-the-possibility-o...

I agree it’s crazy good. But timeline-wise, GPT-3 was in beta and used by many companies in 2020.

Commenters here seem a little fixated on the fact that the technique scores poorly. This is true and somewhat problematic but exploring the data it looks to me like this could be more a problem with a combination of the scoring function and the limits of language rather than the methodology.

For example, look at https://openaipublic.blob.core.windows.net/neuron-explainer/...

It's described as "expressions of completion or success" with a score of 0.38. But going through the examples, they are very consistently a sort of colloquial expression of "completion/success" with a touch of surprise and maybe challenge.

Examples are like: "Nuff said", "voila!", "Mission accomplished", "Game on!", "End of story", "enough said", "nailed it" etc.

If they expressed it as a basket of words instead of a sentence, and could come up with words that express it better I'd score it much higher.

> "This work is part of the third pillar of our approach to alignment research: we want to automate the alignment research work itself."

I feel like this isn't a Yud-approved approach to AI alignment.

Honestly, I think any foundational work on the topic is inherently Yud-favored, compared to the blithe optimism and surface-level analysis at best that is usually applied to the topic.

Ie, I think it's not that this shouldn't be done. This should certainly be done. It's just that so many more things than it should be done before we move forward.

DISCLAIMER: I think Yudkowsky is a serious thinker and his ideas should be taken seriously, regardless of whether not they are correct.

Your comment triggered a random thought: A perfect name for Yudkowsky et al and the AGI doomers is... wait for it... the Yuddites :)

Already used on 4chan :)

These were my thoughts exactly. On one hand, this can enable alignment research to catch up faster. On the other hand, if we are worried about homicidal AI, then putting it in charge of policing itself (and training it to find exploits in a way) is probably not ideal.

"Yud-approved?"

Meaning is approved by Eliezer Yudkowsky.

https://en.wikipedia.org/wiki/Eliezer_Yudkowsky https://twitter.com/ESYudkowsky https://www.youtube.com/watch?v=AaTRHFaaPG8 (Lex Fridman Interview)

He's the one in the fedora who is losing patience that otherwise smart sounding people are seriously considering letting AI police itself https://www.youtube.com/watch?v=41SUp-TRVlg

That's not a fedora, that's his King of the Redditors crown

https://en.m.wikipedia.org/wiki/Eliezer_Yudkowsky

You mean Yudkowski? I saw him on Lex Fridman and he was entirely unconvincing. Why is everyone deferring to a bunch of effective altruism advocates when it comes to AI safety?

I heard him on Lex too, and it seemed to be just a given that AI is going to be deceptive and want to kill us all. I don't think there was a single example of how that could be accomplished given. I'm open to hearing thoughts on this, maybe I'm not creative enough to see the 'obvious' ways this could happen.

This is also why I go into chess matches against 1400 elo players. I cannot conceive of the specific ways in which they will beat me (a 600 elo player), so I have good reason to suspect that I can win.

I'm willing to bet the future of our species on my consistent victory in these types of matches, in fact.

Again, a given that AI is adversarial. Edit: In addition, as an 1100 elo chess player, I can very easily tell you how a 1600 player is going to beat me. The analogy doesn't hold. I'm in good faith asking how AI could destroy humanity. It seems given the confidence people who are scared of AI have in this, that they have some concrete examples in mind.

Other caveman use fire to cook food. Fire scary and hurt. No understand fire. Fire cavemen bad.

IMHO the argument isn't that AI is definitely going to be deceptive and want to kill us all, but rather that if you're 90% sure that AI is going to be just fine, that 10% of existential risk is simply not acceptable, so you should assume that this level of certainty isn't enough and you should act as if AI may be deceptive and may kill us all and take very serious preventive measures even if you're quite certain that it won't be needed - because "quite certain" isn't enough, you want to be at "this is definitely established to not lead to Skynet" level.

And even with all that, probably it's best to still exercise an abundance of caution, because you might have made a mistake somewhere.

Because they have arguments that AI optimists are unable to convincingly address.

Take this blog post for example, which between the lines reads: we don't expect to be able to align these systems ourselves, so instead we're hoping these systems are able to align each other.

Consider me not-very-soothed.

FWIW, there are plenty of AI experts who have been raising alarms as well. Hinton and Christiano, for example.

People won't care until an actually scary AI exists. Will be easy to stop at that point. Or you can just stop research here and hope another country doesn't get one first. Im personally skeptical it will exist. Honestly might be making it worse with the scaremongering coming from uncharismatic AI alignment people.

Are you kidding? "easy to stop"? When LLMs are integrated into law enforcement, banking, research, education, logistics... All these areas have people building backend systems leveraging the current llm tech and pipelines to plug in the coming tech. If we reach a tipping point where these things become aware/act truly autonomously, what are the chances they do it before we notice? People are renowned for implementing things before understanding the consequences.

And what does charisma of AI alignment folks have to do with anything?

Why would it be easy to stop at that point? The believable value prop will increase in lockstep with the believable scare factor, not to mention the (already significant) proliferation out of ultra expensive research orgs into open source repos.

Nuclear weapons proliferated explicitly because they proved their scariness.

> Why is everyone deferring to a bunch of effective altruism advocates when it comes to AI safety?

I'm not sure Yudkowski is an EA, but the EAs want him in their polycule.

He posts on the forum. I'm not sure what more evidence is needed that he's part of it.

https://forum.effectivealtruism.org/users/eliezeryudkowsky

I guess it's true, not just a rationalist but also effective altruist!

Agreed, Yud does seem to have been right about the course things will take but I'm not confident he actually has any solutions to the problem to offer

His solution is a global regulatory regime to ban new large training runs. The tools required to accomplish this are, IMO, out of the question but I will give Yud credit for being honest about them while others who share his viewpoint try to hide the ball.

Good point, but I think if he was serious he would be looking to Serena Butler for inspiration not nuclear non-proliferation treaties

Which things has he been right about and when, if you recall?

At least to me, this current generation of AI progress is looking a lot more "foomy" than a lot of people besides him predicted

Why does this matter?

I think this is a generous usage of "can." As the article admits, these explanations are 'imperfect' and I think that is definitely true.

It depends how you parse it. It is clearly true that they 'can' explain neurons, in the sense that at least some of the neurons are quite well explained. On the other hand, it's also the case that the vast majority of neurons are not well explained at all by this method (or likely any method).

I̵t̵'̵s̵ ̵o̵n̵l̵y̵ ̵b̵e̵c̵a̵u̵s̵e̵ ̵o̵f̵ ̵a̵ ̵q̵u̵i̵r̵k̵ ̵o̵f̵ ̵A̵d̵a̵m̵W̵ ̵t̵h̵a̵t̵ ̵t̵h̵i̵s̵ ̵i̵s̵ ̵p̵o̵s̵s̵i̵b̵l̵e̵ ̵a̵t̵ ̵a̵l̵l̵,̵ ̵i̵f̵ ̵G̵P̵T̵-̵2̵ ̵w̵a̵s̵ ̵t̵r̵a̵i̵n̵e̵d̵ ̵w̵i̵t̵h̵ ̵S̵G̵D̵ ̵a̵l̵m̵o̵s̵t̵ ̵n̵o̵ ̵n̵e̵u̵r̵o̵n̵s̵ ̵w̵o̵u̵l̵d̵ ̵b̵e̵ ̵i̵n̵t̵e̵r̵p̵r̵e̵t̵a̵b̵l̵e̵.̵

EDIT: This last part isn't true. I think they are only looking at the intermediate layer of the FFN which does have a privileged basis.

> EDIT: This last part isn't true. I think they are only looking at the intermediate layer of the FFN which does have a privileged basis.

it does?

Yeah, that's where they apply the activation function and that happens per neuron so you can't rotate everything and expect the same result.

I'm split between what's more impressive:

- The software powering the research paper

- The research itself (holy moly! They're showing the neurons!)

This blog post is not very informative. How did they prompt GPT4 to explain the neuron's behavior?

It's explained more in the "read paper" link, where they provide the actual prompts:

https://openaipublic.blob.core.windows.net/neuron-explainer/...

Language models can also tell you they’re not AI.

With dubious confidence too!

IMO this is akin to how scientists developed thermodynamics to comprehend steam engines. Except now, AI could be on the forefront of forming conjectures and explanatory theories!

We don’t really know how harmful this stuff will be, but we’ll keep on making it more powerful (and keep on bragging about it) until we find out.

Even if we can explain the function of a single neuron what do we gain? If the goal is to reason about safety of computer vision in automated driving as an example, we would need to understand the system as a whole. The whole point of neural networks is to solve nuanced problems we can't clearly define. The fuzziness of the problems those systems solve is fundamentally at odds with the intent to reason about them.

> reason about safety of computer vision in automated driving

An interesting analogue. I think we simply aren't going to reason about the internals of neural networks to analyze safety for driving, we're just going to measure safety empirically. This will make many people very upset but it's the best we can do, and probably good enough.

I have to agree.

I often think, “maybe I should use ChatGPT for this” then I realise I have very little way to verify what it tells me and as someone working in engineering, If I don’t understand the black box, I just can’t do it.

I’m attracted to open source, because I can look at the code understand it.

Open source doesn't mean you can explain the black box any better. and humans are black boxes that don't understand their mental processes either. We're currently better than LLMs at it i suppose but we're still very poor at it.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522

We can't recreate previous mental states, we just do a pretty good job (usually) of rationalizing decisions after the fact.

Humans can be held accountable so it’s not the same. Even if we’re a black box, we share common traits with other humans. We’re trained in similar ways. So we mostly understand what we will and won’t do.

I think this constant degradation of humans is really foolish and harmful personally. “We’re just black boxes etc”, we might not know how brains work but we do and can understand each other.

On the other hand I’m starting to feel like “AI researchers” are the greatest black box I’ve ever seen, the logic of what they’re trying to create and their hopes for it really baffle me.

By the way, I have infinitely more hope of understanding an open source black box compared to a closed source one?

>Humans can be held accountable so it’s not the same.

1. Don't worry, LLMs will be held accountable eventually. There's only so much embodiment and unsupervised tool control we can grant machines before personhood is in the best interests of everybody. May be forced like all the times in the past but it'll happen.

2. not every use case cares about accountability

3. accountability can be shifted. we have experience.

>I think this constant degradation of humans is really foolish and harmful personally.

Maybe you think so but there's nothing degrading about it. We are black boxes that poorly understand how said box actually works even if we like to believe otherwise. Don't know what's degrading about stating truth that's been backed by multiple studies.

Degrading is calling an achievement we hold people in high regard who accomplish stupid because a machine can do it.

>By the way, I have infinitely more hope of understanding an open source black box compared to a closed source one?

Sure i guess so.

If there is a limited number of deception neurons, we can automatically look for coactivation of the "my future behavior" neuron and the deception neuron.

My take: Regurgitation of trained-on information about LLMs does not come anywhere close to "conscious brain knows it's conscious."

Has anyone here found a link to the actual paper? If I click on 'paper', I only see what seems to be an awkward HTML version.

You mean this? https://openaipublic.blob.core.windows.net/neuron-explainer/...

Would you prefer a PDF?

(I'm always fascinated to hear from people who would rather read a PDF than a web-native paper like this one, especially given that web papers are actually readable on mobile devices. Do you do all of your reading on a laptop?)

With a pdf I don't have to update my PDF reader multiple times per month just to be able to read text.

A PDF is a text document that includes all the text, images, etc within it in the state you are going to perceive them. That web page is just barely even a document. None of it's contents are natively within it, it all requires executing remote code which pulls down more remote code to run just to get the actual text and images to display... which they don't in my browser. I just see an index with links that don't work and the the "Contributions" which for some reason was actually included as text.

Even as the web goes up it's own asshole in terms of recursive serial loading of javascript/json/whatever from unrelated domains and abandons all backwards compatibility, PDF, as a document, remains readable. I wish the web was still hyperlinked documents. The "application" web sucks for accessibility.

The equations look terrible on Firefox for Android, as they are really small - a two-line fraction is barely taller than a single line, forcing me to constantly zoom in and out.

So yes, I would prefer a PDF and have a guarantee that it will look the same no matter where I read it.

> Would you prefer a PDF?

Yes, I was just reading the paper and some of the javascript glitched and deleted all the contents of the document except the last section, making me lose all context and focus. Doesn't really happen with PDF files.

> always fascinated

That feels like a loaded phrase. Is it "false confusion" adjacent?

My whole workflow of organizing and reading papers is centered on PDFs. While I like having interactive supplemental materials, I want to be able to print, save and annotate the papers I read.

If you want to draw on it, PDF is usually the best

Nope, reading the printed paper on... paper. :)

I personally prefer reading PDF on an iPad so I can mark it up.

How long until we copyright the Marvel Neuron?

I’m most surprised by the approach they take of passing GPT tuples of (token, importance) and having the model reliably figure out the patterns.

Nothing would suggest this should work in practice, yet it just… does. In more or less zero shot. With a completely different underlying model. That’s fascinating.

They’re not looking at activations?

No just at normalized log probabilities I think

At some point did we change the name from perceptron to neuron? Neural networks don't have neurons.

More in detail: https://openaipublic.blob.core.windows.net/neuron-explainer/...

Now grab the list of labels/explanations for each neuron, and train a small LLM only with data for that neuron.

Then you get a dictionary/index of LLMs

Could this be used to parallelize training?

Or create lighter overall language models?

The above would be like doing a “map”, how would we do a “reduce”?

I’m so interested in this. Any ideas how I can get involved with only a 2014 laptop?

https://www.w3schools.com/ai/default.asp

Wow— this page is superb.

https://www.neelnanda.io/mechanistic-interpretability/gettin...

By analyzing the function of individual neurons, these guys were/are able to gain a deeper understanding of how language models process language, which could lead to improved model architecture and training methods.

OpenAI need to hear an explanation of the word "explain".

This is really interesting. Could this lead to eventually being able to deconstruct these "black-boxes" to remove proprietary data or enforce legal issues?

Reminds me of the boo Blink, by Malcolm Gladwell -- We feel like we have to come up with a reason why we came up with an answer (paraphrasing)

I wish there was insightful explanations on why AI cannot think, and if there are researchers trying to explore this topic, and if yes what they do.

Any discussion on this topic would be nothing but arguments over the definition of the word 'think'.

Can anyone explain what they did? I’m not understanding from the webpage or the paper. What role does gpt4 play?

I’m seeing they had gpt4 label every neuron but how?

I got the impression that it mentioned that the complexity of what's going on in GPT is so complex that we should use GPT to explain/summarize/graph what is going on.

We should ask AI, how are you doing this?

Operator: Skynet, are you doing good thing? Skynet: Yes.

Automating safeguards and interpretability decisions seems circular and likely to detach to policy.

its interesting that layer 0 is a bunch of different things like upper case letters and symbols and types of verbs.

it would be great to see all the things theyve found for different layers

Can't wait for West World style sex robots!

Matrices, all the way down

LLMs are a strange loop.

Can't see how using AI to explain AI could ever go wrong

Skynet anybody?

Great. Going meta with an introspective feedback loop.

Let's see if that's the last requisite for exponential AGI growth...

Singoolaretee here we go..............

There is no introspection here.

    ...our approach to alignment research: we want to automate the alignment research work itself. A promising aspect of this approach is that it scales with the pace of AI development. As future models become increasingly intelligent and helpful as assistants, we will find better explanations.

The distance between "better explanations" and using that as input of prompts that would automate self-improve is very small, yes?

Reply on news.ycombinator.com