Comment by YeGoblynQueenne | Hacker News Reader

Comment by YeGoblynQueenne | Hacker News Reader

YeGoblynQueenne

>> One trend that continues from the common sense is that GPT-3 is reluctant to express that it doesn’t know the answer. So invalid questions get wrong answers.

More to the point, it's not that GPT-3 knows any answers to any questions. Like the article says, language models are trained to predict the next characters in a sequence. So, it predicts that a sequence that starts with "Who was president of the United States in 1700?" would continue with the sequence "William Penn was president of the United States in 1700". Seen another way, there's a correlation between the two strings. It's the highest correlation between the first string and any other string, so that's what ends up in the output.

So it's more accurate to say that GPT-3 is guessing what the answer to a question might look like, than to say that it "knows" what the answer to a question, is. It's perhaps a subtle distinction but it's important to keep it in mind, especially in view of claims about "intelligence" and (Old Ones save us) "understanding".

Basically, what all this tells us is that it's not possible to answer arbitrary questions consistently by guessing at them. Even if you get lucky, and you keep getting lucky, there will always be questions that you can't answer correctly and they will be many more than the questions you can answer correctly (e.g. we could generate an infinite number of common nonsense questions like how to sporgle a morgle, or who was the president of the united states since Biblical times etc).

This of course is an old debate in AI: can we simulate a system of logic, without implementing its logic? e.g. could we perform correct arithmetic by memorising the results of calculations? Can prediction replace reasoning? Despite the unending hype generated by OpenAI about its models, the answer keeps being: a resounding no. While you can go a long way by training on ever larger data, it only takes a shallow search before obvious nonsense results are generated.

If prediction cannot replace reasoning, then how would you define reasoning? If reasoning is the process of inference of the most likely from the most similar known, then how does the multi-head attention transformer not fit that description?

>> If reasoning is the process of inference of the most likely from the most similar known, then how does the multi-head attention transformer not fit that description?

I don't know, because I did not propose that definition of reasoning.

"Reasoning" does not have a formal definition in AI, or Computer Science (neither does "intelligence" or "understanding") but, in general, when we speak of reasoning in those fields we mean a procedure that can derive the logical consequences of some premises, often in conjunction with some background theory. I'm happy to use this as an informal definition of reasoning, if you want.

Okay. But how do we determine if something is logical? At the very least we have to abstractly infer and compare with what we have been taught is logic (because hardcoding it manually doesn't map to the [hierarchical] intricacies of reality very well). So logic is a high level reasoning which has to be powered by the low-level abstract/infer/mimic as exhibited e.g. by the transformer.

I wonder if expert systems could be used to generate a "logical reasoning" training dataset (or a cost/fitness function) to help train/evolve neural networks on. Or if there are other ways of integrating these two.

Ah, now logic is something that has a very clear formal definition- actually, many, because there are many different logics. For example, there's propositional logic, first order logic, temporal logic, description logic, default logic, etc etc.

So, how do you decide whether something is logical? In particular, if we have some system S that exhibits behaviour H in context B, can we tell whetehr H is, in some sense, "logical"? Why, yes we can: we can apply the rules of whatever logic we think S may be following with H and see if we can reproduce H in the context of B, starting from the same premises as S.

For example, if S is producing a behaviour H that can be described as {A → B ≡ B → A} and we think that S is trying to reproduce propositional logic, we can say that it's failing, because H is not correct by the rules of propositional logic (to clarify, H could be something like "If it rains it will be wet therefore if it is wet, it has rained", which is not correct).

Of course the problem with GPT-3 and friends is that they are not trained to output statements in a formal language (at least not exclusively) so it's very hard to formalise the logic, or lack thereof, of its output. Though it would be interesting to "hit" GPT-3 with some prompts representing the beginning of natural language versions of formal problems.

Could expert systems be used to generate training data for GPT-3? Maybe. If an expert system could generate good quality natural language, even only natural language restricted to some limited domain. Then yes, why not? You could probably train GPT-3 on its own output, or that of its predecessors, as long as it was curated to remove nonsense (which is much harder than it sounds).

Reply on news.ycombinator.com