Comment by js8 | Hacker News Reader

Comment by js8 | Hacker News Reader

I have an objection to the idea that understanding is just a pattern recognition, or prediction of the next token. I think fundamental to understanding is the concept of logical contradiction, which I don't believe GPT-3 has. I don't think GPT-3 can say, wait, what I am saying is BS, because logically it's not consistent.

To argue that, I propose a "simplified Chinese room". We have an automaton in the Chinese room, which does only pattern recognition, and can learn by matching patterns and structures, i.e. it will give an answer that is similar to what it has seen before. And we want to teach it to answer a problem "recognize whether the given logical formula is satisfiable". So we give it formulas and it tries to pattern match the formulas it already has seen. Unfortunately, there always exists a formula which is matching closely everything that was shown so far, and yet the correct answer is unexpected. So such an automaton will never be able to discover the actual rule.

What I think has to be added are different levels of discourse, or, a discourse about discourse. With people, we can say, "stop joking around", and it's a meta-discourse signal that we want stop discussing imaginary worlds and focus on the real one, for example, which has to be totally self-consistent. I believe an analogy can be made with Futamura projections, we can have not only systems that learn a specific task (like even GPT-3 does), but also systems that learn to produce an agent to do a specific task, from a metalanguage description that encodes certain rules on how to do it (for example, to follow logical consistency). And if the metalanguage is general enough, you get general intelligence, because then you can apply the agent production process to itself.

> I think fundamental to understanding is the concept of logical contradiction, which I don't believe GPT-3 has

are we sure that humans actually have that though ? It's not always my experience

I think we do. We get a distinct feeling that we don't understand something, and a different feeling when things click together. Or when we hear a syntactically valid (in other words, pattern-matching) but nonsensical sentence (like "colorless green ideas sleep furiously"), or see a Penrose triangle, and we cannot imagine an example, we get a very specific feeling that something is wrong. I think we are actually solving the satisfiability problem every time we imagine something - we find a concrete instance of a situation that satisfies certain conditions (for example, words in a sentence, or pixels in the image).

The book Inside Jokes even speculates that reason why we find jokes pleasurable is that they are actually a learning mechanism to detect logical inconsistencies and further our understanding of the world.

With GPT-3, you can't tell whether it's joking or not, and whether it's aware if it's joking or not. It doesn't (to my knowledge) have that boundary where it could say, wait, I don't understand what we are saying here. Maybe it's easy to add to the system, but as it stands, I don't think GPT-3 does the above.

> With GPT-3, you can't tell whether it's joking or not, and whether it's aware if it's joking or not

But you can't genuinely know that for humans either. At best you can they that they look like they are joking, and you definitely cannot say that they in fact are, even if they say so to your face.

Well, I think you sorta can. Your boss can say, if you don't stop joking around, and don't follow the rules, I will shoot you. We have an evolutionary predisposition to avoid pain and death and the whole social structure on top of that, and this ensures that sometimes we just have to be honest.

Which is one of the reason why I think we need at least two levels of learning for general intelligence. The bottom learner is the one which learns the actual task at hand, and the top learner is tasked (with learning) to, given a description of a task, produce (train if needed) a bottom learner that can solve that task. This solves the ambiguity, because if we are confident enough that the top learner can produce correct bottom learners for different descriptions, then adding another rule to the description will likely cause the bottom learner (produced by the top learner) to also obey that rule.

It also resolves the understanding problem, because while no learner might be sure if it is itself correct, the top learner has to somehow evaluate the bottom learner, and so the top learner sees where bottom learner is failing to understand (which rules are problematic). What's even more interesting, if the language to express the problems is rich enough, then we can apply the top learner to itself as the bottom learner (express the actual learning setup from within the language), and we get sort of a "philosopher", entity that tries to learn how best to build learners. (This is an interesting analogy you can make with Futamura projection, by thinking about "learners" instead of "programs", and what happens if learners are tasked to produce another learner. The reason why we can be somewhat confident in "correctness" of the bottom learners is similar to reason why we can be somewhat confident that compiled program does what we want, despite it being completely new to the compiler.)

There are multiple ways to map this top/bottom learners onto humans. You can think of the bottom learner as the actual intuitive systems that we have, and the top learner as the conscious system, which drives what we are gonna learn and how. Or the bottom learner as the system that contains the model of the environment, and the top learner as a system that provides episodic learning via some kind of simulation (our dreams, imagination) to it, and reflects on it. Or the bottom learners are individual humans, while top learners are the genes that drive the evolution of the human brain. In practice, there might be more than two levels.

> With people, we can say, "stop joking around", and it's a meta-discourse signal that we want stop discussing imaginary worlds and focus on the real one, for example, which has to be totally self-consistent.

-

> We get a distinct feeling that we don't understand something, and a different feeling when things click together. Or when we hear a syntactically valid (in other words, pattern-matching) but nonsensical sentence (like "colorless green ideas sleep furiously"), or see a Penrose triangle, and we cannot imagine an example, we get a very specific feeling that something is wrong.

It's true that the human mind has the capability to do these things sometimes (and within certain specifically designed frameworks like the scientific method, do it highly reliably)....but whether we do do it (or even have the capability) with consistent, high level success across various domains is a very different matter.

> prediction of the next token

Assessing logical consistency of a following statement would increase prediction accuracy (assuming that human-generated text is more consistent than it is not), but only after you have plucked all lower-hanging fruit. The best model of what comes next in a math paper is a mathematician.

So GPT-N not trying to does not necessarily mean GPT-N+1 can't do it, it just means that it has cheaper ways to increase its prediction scores.

This raises a question whether the best prediction should be a measurement of intelligence. GPT-3 might well be capable of super-intelligence, for what we know, but if we feed it examples on human level of thought, it will interact with us on human (i.e. stupid) level of thought, because that's what we are teaching it. Just like adults choose to talk to children on their level.

However, a truly intelligent system, I think, can transgress this boundary. It can protest against the task of giving the next best prediction, so to speak.

I don't think that mathematician is necessarily the best model for a math paper; I think the best model might as well be the platonic ideal of what it means to do mathematics - logical correctness of all the conclusions.

Unless you're substituting GPT3 for AGI, GPT3 is not a super-intelligence. It's a set number of matrix multiplications that completes in polynomial time. Because Transformer context is fixed, and given the limitations of training, adding non-determinism (tree search) means it goes off the rails and is prone to getting trapped in cycles anytime it doesn't quickly terminate.

Has anyone tried prompting GPT-3 with "the following is a transcript of a conversation with a superintelligent artificial intelligence, in which it solves all of humanity's problems"?

Reply on news.ycombinator.com