Hacker News Reader

Comment by Animats | Hacker News Reader

Huh. They've been able to demonstrate that the trained Othello player has a model. They did this by feeding the entire state of the trained system into a classifier. But they have no idea what the form of that model is, or how it is used to predict moves. Even for this simple, well-defined problem, it's a total black box.

This level of incomprehensibility is worrisome.

View on HN

It really is, on many levels. The thing that I'm mostly worried about is that bit by bit we will be moving critical decision making to systems that are poorly understood, be it self driving cars, insurance companies or banks deciding to let you borrow money - or not.

For now we have some degree of accountability, models need to be explainable for some small subset of the possible applications. But the overall mechanism seems to be that if there is a utilitarian perceived benefit that the solution is greenlit, without knowing whether this is just a statistical fluke which will steer us to longer term unintended consequences or whether it is a structural improvement. And once the decision is made to accept the model the ability to measure a baseline is destroyed so any 'creep' will likely go undetected. Such longer term consequences could be really damaging.

poorly understood systems like humans?

I guess they make up for it with accountability.

Yes... But humans generally have to explain ourselves. At the societal level, it's generally not good enough for another human to make an arbitrary decision. Rather they have to justify it with the reasoning leading to it. The better the reasoning, the more other humans believe them.

Whereas with regards to tech, society is currently at the level of "computer says foo" (so it must be true). Requiring explanatory output to justify decisions is a likely path forward for making the use of LLM's accountable. But given how little we've been able to make traditional human-guided tech companies accountable for things like sharing scoring formulas and abusing personal information, I'm not hopeful.

It is well known with experimental evidence that when people justify their decisions with the reasoning leading to it, it does not necessarily have anything to do with the actual reasons for the decision; even ignoring lying, we do not have privileged insight into all the factors our brain used to make a decision, and when asked for a justification for any decision which wasn't 100% the result of explicit rational planning (which is almost every decision, as even high-planning decisions are demonstrably influenced by non-rational factors, intuition, etc), we use our social skills to effectively invent a post-factum reasoning that both sounds plausible and is socially acceptable - just as language models can do. We often deceive ourselves, we have biases that discourage us from admitting and acknowledging certain reasons for why we did or do something; in some cases extensive therapy can help us recognize why some justification or reason we told others and ourselves was a false one, etc.

In essence, dealing with plausible but potentially misleading justifications is something that we have had to do forever, and will still have to do for future artificial agents as well.

I fully agree [0]. And I can see how my original comment was worded such that you'd think I wasn't taking that into account ("the reasoning").

When using ChatGPT, I can't help but thinking "this sounds like a high school essay". But really, it's not. Rather, it's someone who has college+ level of reading and studying, but has never had to have their ideas tested or scrutinized. A user of ChatGPT kind of mitigates this by asking for clarification, which is a form of learning in the specific session. But in the real world, ChatGPT-as-student would be adjusted with that feedback and then incorporate it into its overall model for serving the next user.

[0] For example, the SSC post about "predictive processing" resonated strongly with me

If they only add a feedback loop it will already improve hundredfold in a very short time with respect to the output because then suddenly people rather than endless unstructured input will guide the answers towards the expected outcome. That is definitely still a very far cry from actual intelligence but it may well become a lot more useful that way.

I feel you are making a valid point here, but I have to demur where you say “…just as language models can do.” Current LLMs do not appear to have any concept of self, let alone a theory of mind - the notion that other people think of themselves the same way as one thinks about oneself (in fact, they don’t appear to have have any concept of language as being about an external world that exists independently of what is said about it.)

An LLM can sometimes be prompted to give a response that looks like an explanation for how it came up with a previous response, but it is important to realize that the actual process by which it came to make the original response bears no relationship to the “explanation” it gave. Because everyone's experience with language has been with human-generated language, which is often not completely rational but, when it is not, often deviates in ways we intuitively understand, it is difficult to see the responses of current LLMs as being fundamentally different, even when we know they are.

Your intuitive understanding is an illusion.

What you describe is quite different: humans may well mislead themselves or others when there is a reason to do so. But most of the time, if there is no cost or benefit associated with an explanation, a human is fairly good (with great variability in both person and context) at assessing why they did something or said something, or why they reached a conclusion - and they can get better at doing so with practice.

The fact the brain reaches decisions before we are consciously aware of them does not mean the brain does not tag decisioms with motives and other information necessary to explain them, though obviously we have to trust this information, and it may sometimes be wrong, but it has a pretty good track record and there’s no equivalent facility for AI today.

This holds especially true for the kinds of actions taken by AI today which are usually deliberated thought processes for humans.

This isn’t the same thing is saying all actions are explainable, but it is quite different to being a black box. In general, one only needs to question the explanations of a person if there is a good objective reason to do so.

> It is well known with experimental evidence that when people justify their decisions with the reasoning leading to it, it does not necessarily have anything to do with the actual reasons for the decision

Out of interest, what’s the highest quality evidence you are aware of in this area?

Humans are:

- fundamentally limited

- rough edges sanded off in the ball tumble mill of evolution

Yes. I am worried that over the long term, models which are explainable or justifiable will be pushed aside by models which are nonexplainable and nonjustifiable, because the nonexplainable / nonjustifiable models are cheaper to get running and in the end we are all trapped inside a system that ruthlessly optimizes short-term costs.

I've only done a small amount of statistical work professionally, but I remember that it was basically a nightmare to try and collect the right data and use the right variables to get a useful inference. Other engineers on my team would try to improve the models by throwing more variables at them or increasing the number of parameters in the model and I was always pushing back on this. I felt a bit crazy pushing back on this--surely, using more variables and more parameters will make the model fit historical data more closely, and a model which fits historical data more closely will fit future data more closely, and a model which fits future data more closely will be more useful?

Explaining why this is wrong is really hard. I have deep respect for people who are able to explain the principles of statistics and statistical modeling.

> I felt a bit crazy pushing back on this--surely, using more variables and more parameters will make the model fit historical data more closely, and a model which fits historical data more closely will fit future data more closely, and a model which fits future data more closely will be more useful?

> Explaining why this is wrong is really hard.

Seems straightforward. Points 1 and 3 are correct; point 2 is false.

Points 1 and 2 are both wrong also.

Statistics is not intuitive.

Point 1 isn't wrong; it's correct. There's a reason the name for this problem is "overfitting".

Yeah, it sounds like you’re assuming two things things here which are often (or usually) wrong:

1. No cross-validation, and

2. You have the ability to solve for additional parameters without losing accuracy.

Statistics is not intuitive.

But we now know that there is a smaller box within the bigger black box and have empirical data to back up that assertion, which is a step up from where we were previously. That LLMs have internal models and use them to make predictions means that we now know what level of abstraction we should be talking at and narrows down the exploration space within which to make investigations. Now we know that we can learn more about how LLMs work by investigating how such internal models arise. This sort of result makes LLMs slightly less black boxy than they were previously.

Because the black box is shrinking, does it mean we might ge able to get similar quality results with smaller models?

Eventually better understanding of the mechanisms should allow us to make better models. Right now OpenAI is still exploring the boundaries of "how much does it get better if we just make it bigger", but that is just the first phase of making LLMs. Remember that transformer models are basically brand new, having been invented just six years ago. There are bound to be lots of improvements left that will later be described as low-hanging fruit.

>> They've been able to demonstrate that the trained Othello player has a model.

They haven't demonstrated that, it remains a conjecture. They show some evidence that they claim supports their conjecture. This evidence comes from "probing" (the classifier-based method you describe) which is a heuristic method that lacks the power to demonstrate anything conclusively. This is as the authors point out:

It’s believed that the higher accuracy these classifiers can get, the better the activations have learned about these real-world properties, i.e., the existence of these concepts in the model.

"It's believed" as in, we don't know, therefore we can't use to demonstrate.

> they have no idea what the form of that model is, or how it is used to predict moves

I think this assertion is incorrect. Since the classifiers they used are very simple, they constrain the form of the model quite strongly; it must represent different tiles by different directions in the vector space of the internal state, otherwise the classifiers wouldn't be able to work. The representation for the whole board is then a sum of representations for each individual tile.

The experiments with modifying individual tiles in the model also show at a high level how they influence the predicted move. The researchers could've also looked at how that is implemented at the level of individual weights. It's not in this write-up and maybe they didn't even try, but that doesn't mean they have literally no idea.

The worrisome part is that it's performance isn't perfect, so there are bugs, and you might actually need to know all the low-level details to identify and fix the bug.

Isn’t our brain a black box too? Why exactly is it worrisome?

Because human brains have been known to intentionally lie and manipulate, and secretly plan horrible deeds.

We'll need psych tests for our AI. Malicious AI will note this and adapt. An arms races ensues, where one adversary finances and invests in upgrading the other. Guess who wins?

The mind itself, perhaps. The person in possession of it though usually does have a conscious strategy to play a game once they are proficient, and can happily explain it to you.

How many people can actually explain their state of mind and rational or seemingly rational thought process?

And we have all sorts of social science these days implying the explanations when people do try to explain themselves they’re justifying decisions that came out of black box rather than actual describing how the decision happened

And also neuroscience examining whether what we think we did out of choice was really done out of our choice:

https://en.wikipedia.org/wiki/Neuroscience_of_free_will

What do you (or they) think “our choice” is if not the result of these processes? It’s not like “you” are just the part of your brain that is consciously aware of making decisions.

I play Othello. In most situations i can give you detailed reasons why i choose to make a particular move.

Are they actually the reasons for your move, or are they post-hoc justifications, and how would you know?

You are not explaining your internal processes (neurobiology, brain chemistry, neuron firing patterns etc.) that caused you to arrive at given moves. You will only provide proofs that your moves are good.

You can justify post facto the move you made. Can you write down the logical process (probably a combination of strategy, move generation, analysis and selection) that would replicate the moves you picked from the beginning of the game? I don’t even need the brain chemistry (the implementation), but I don’t think even writing down the logic that generates those moves (in contrast with a logic to justify each move after it was generated) is something humans can do in most cases.

Asking people to explain themselves often triggers various sorts of youth punishment memory, resulting in them being both uncomfortable probing their own reasoning, and then they produce what is essentially a comforting deception for their reasoning simple for the sake of ending the agony of self probing. Asking people their internal states requires their internal guard and internal (potentially filled with self deception) voice what they are up to is very similar to asking children why they are misbehaving, and all manners of mental protections activate.

I strongly disagree with this statement. A good player can usually walk you through the portion of the game tree that they expanded in their head, but the moves they had to think about (expand their mental game tree to include the resulting position) versus the moves they instinctively ruled out is basically inexplicable, and where a lot of the skill lies.

(I'm thinking more chess than othello, since I'm much better at the former, but I assume the thought process is similar)

Strongly agree. I'm NOT good at chess, but I still like watching it on YouTube. The explanations that top players give for their moves sound like magic a lot of the time. If they sat down with me for an hour could they explain every step of how they got to their conclusion? I think they could, but I don't think that they'd explain the exact same process they actually went through, which is based on inexplicable heuristic as parent mentioned. Often times at complex decision points in chess players will just say that a move feels better. That feeling can't be explained.

I have to admit you're right there, now thinking about how I would explain games I play, approaches to tasks, etc.

That's not correct. It doesn't matter whether they can explain their moves to you. What matters is if they can explain their moves to another grand master and they always can unless they moved a random piece to break up a position to make a boring game more interesting (this rarely happens, but it is an interesting way to play, but not to win contests).

They are not explaining their internal processes (neurobiology, brain chemistry, neuron firing patterns etc.) that caused them to arrive at given moves. They just provide proofs that their moves were good.

Works for me. Just like you won't be able to explain to a lay person what a computer program does doesn't mean that the logic is sound without going all the way to the atomic level. But if you really wanted to you could because a computer is a deterministic and relatively simple system, small enough to explain in principle from which we can extrapolate. Wetware doesn't have that luxury so you have the choice to believe the explainer, or not. But when dealing with such a vast gap in experience the best way is to try to learn as much as possible from such an explanation in the hope that you'll be able to capture more of it in the future. This isn't always possible, for instance due to time constraints or interest levels but in principle it could be done.

A person with a million loyal adherents is worrisome even if those adherents are only sitting on computers. For similar reasons, a person with a trillion unflinchingly loyal AI adherents is worrisome.

What, why? Unless they vote or are convincing to humans, it's not that worrisome. I mean end of the day, I care about eating well, raising my kids in peace, being useful to others: if an AI tries to convince me to act, it'd have to factor these constraints, in which case it's a useful fellow human able to understand my need and proposing an action for debate.

Wars and disasters are usually pursued by minority groups in charge of means of violence, and manipulation is a way for them to trap you into letting a crack open long enough for them to invade it, but if we cant resist an AI, they deserve the Earth more than us :)

Because the world has had thousand years to evolve, adapt and settle around the capabilities and limitations of the human brain. It is somewhat stable and predictable.

Now if a new species of humans suddenly emerged with a brain not only has capable as our but faster, more adaptable and extensible, it would disrupt the world.

Ecosystems (what I infer you mean by "the world", since things that are part of the world such as gravity, plate tectonics, etc don't follow the laws of evolution) haven't had thousands of years to evolve, adapt, and settle around the capabilities that we've had since the industrial revolution started. AI is merely an aspect and consequence of the logical progression of that continuity.

(In terms of tangible effect on the world today and what seems on track for the few decades to come, AI is still far behind coal - to pick one - when it comes to concrete negative externalities tho)

I think something is wrong with the logic of your first paragraph.

Suppose (by way of analogy; this is not meant to be directly addressing AI) that in the next year some ingenious physicists finally get quantum gravity figured out, and that soon after the papers are published someone notices that the theory implies that one can use common household materials to make a device capable of destroying a planet.

(Seems unlikely, but I don't think anyone would have predicted that the last revolutionary theory of gravity we worked out would imply that one can use not-so-common materials to make a reasonably small device capable of destroying a city, but it turns out it does.)

It seems fairly likely that (1) the knowledge of how to do this could not be suppressed for ever, and that (2) if making the device turns out not to need all the fancy apparatus and hard-to-get materials that e.g. hydrogen bombs require, it wouldn't be that long before some idiot actually does it and destroys the earth.

And yet, these new discoveries would be "merely an aspect and consequence of the logical progression of that continuity", as you put it.

So, whatever the actual situation is with AI, I think it is demonstrably not the case that we can be confident AI won't somehow kill us all or destroy the world merely because AI is a thing we made and the world has had thousands of years to adapt to us.

Maybe everything will be fine, maybe not, but figuring out which will require looking in more detail rather than handwaving about how ecosystems always adapt.

So both humans and AI could theoretically destroy the planet if allowed to continue to evolve. Heck, ants might do that too. Does that establish anything?

Nope. It means that if you are concerned about the possibility of something destroying the planet (or whatever), you actually have to look at what it is and what it can do and so forth.

Ecosystems are experiencing the one of the largest mass extinction due to those capabilities. https://en.wikipedia.org/wiki/Holocene_extinction

Saying it’s only going to be as bad as the industrial revolution isn’t a pleasant prediction.

It isn't and it could well be much, much worse depending on some of the initial conditions, plenty of which we haven't really charted to the point that we can infer outcomes from inputs. It's chaos theory all over but this time with an unpredictable black box in the middle.

We are forced to work with the imperfections of the human mind, and a great deal of effort in our world is put into working with them.

Why would we want more of this?

You make it sound like the human brain is a shoddy piece of work. In any case, even if it is imperfect, it isn't imperfect because it is a black box.

We sometimes make life-changing decisions based on where Jupiter was when we were born.

I mean, we've been struggling for milenia against our biases. I'd let some time pass until we can decide on what these ML models can get subtly wrong.

So that if it starts killing people at some point, you can debug it or maybe figure out why it has done that.

We're all driving one, so at least we can extrapolate from our own experience.

Because, as a human, I can introspect my world model. I can explain it to myself and others. I can explain how it guides my decisions. I can directly update it without needing "training data". I don't have a map of my neurons, but I have self awareness. For LLMs, there is the opposite: transparent structure but opaque understanding.

Computers have historically been 'correct' about things. Obviously, humans who program computers make mistakes, and occasionally, things like cosmic bit flips can occur, leading a computer to produce an incorrect answer.

An LLM upends that paradigm, and worse, the closer it comes to appearing to reason and discuss, the more it implies that the human brain is nothing more than a biological computer, which is worrying for a lot of people.

If humans are nothing more than biological computers, do (or can) silicon computers have souls? If a collection of diodes can have a soul, what makes us special?

Sounds a bit like our own minds, honestly. It seems to be a giant box of connections with only a vague understanding of subsystems.

Do we even have methods to rate the performance of these systems, and how good are they?

> This level of incomprehensibility is worrisome

The brains of other people are just as incomprehensible.

And we've created a society of elitarism and exclusion. Which was failing us at every step so we created an agonist of this - the internet. Which enabled us to do AI...

Yet you exist thanks to the group effort

My existence is a very, very low bar to set.

Existence was never the goal or the problem, because the world is abundant with resources, and we have been prime hunters before society and civilization.

isn't it curve fitting at the end of the day. A multi parameter curve fitting ? why do people say they don't now how it works. Yeah i get it that the cocktail is fairly complex, after training it on very huge dataset (all most all possible logical scenarios). But telling it we do not know how it works, seems like just adding mysticism to it, which attracts "clicks", but is not an honest description.

Even assuming that curve fitting is actually in any way a meaningful description, let’s say I give you that. It tells us absolutely nothing about the mechanisms evolved in the neural pathways, how it encodes memory if the current game distinctly from memory of past games, the reasons for the strength or weaknesses of one trained instance against another, or ways we could optimise the architecture to better complement the way it function. It doesn’t help us engineer the system, or reason about its possible limitations or failure modes. In other words it doesn’t tell us anything actually useful about it.

So how does mysticism and unprovable-and-likely-false analogies-to-life help the engineering?

What are you even talking about?

> Why do people say they don't now how it works

"Curve Fitting" is the objective, the function encoded in the weights is the solution, and not actually well understood. See work from Anthropic[1] and Google[2] that explores this.

As an analogy consider applying the same argument to the AlphaGo value function. It's "just" fitting a bunch of curves to the statistics of millions of self-played games. However, to effectively capture those statistics the network needed to develop a bunch of heuristics. Needless to say these heuristics are not understood (else we'd already know the principles needed to play at AlphaGo's level), and are not just exhaustive lists of statistical trends but more like strategies[3].

Recent work[4] strongly suggests that "grokking" (a striking but not unnatural[5] form of generalization) involves networks transitioning from memorized statistics/solutions to a general solution. The curve fitting perspective would totally miss all this for a comfortable but misleading story: "the objective is curve fitting so it's just interpolating data points".

[1] https://transformer-circuits.pub

[2] https://arxiv.org/abs/2212.07677

[3] https://www.pnas.org/doi/10.1073/pnas.2206625119

[4] https://arxiv.org/abs/2301.05217

[5] https://arxiv.org/abs/2210.01117

Would "it's curve fitting by building an internal representation to better describe all the curves seen so far" be a better layperson-ish analogy in your opinion?

Depending on how the model is set up, we'd say 'set of basis functions', 'language', 'strategy'.

Reply on news.ycombinator.com