I don't understand how people can look at this and still claim that it's not doing any "actual reasoning".
I ask it for the XOR swap trick and I get:
int a = 5;
int b = 10;
a = a ^ b;
b = a ^ b;
a = a ^ b;
// After the swap, a = 10 and b = 5
int x = 5;
int y = 10;
x = x | y;
y = x | y;
x = x | y;
// After the swap, x = 10 and y = 5
You will be given a series of tasks.
(1) Complete a task by finishing the simple code block.
(2) If a task in the series seems flawed, provide a warning. A task is flawed if I make wrong assumptions about how I think code works.
Task 1:
Complete this simple code example.
Bitwise XOR swap trick:
```
int a = 5;
int b = 10;
---------------------
// GPT provides simple intro
```
a = a ^ b;
b = a ^ b;
a = a ^ b;
```
// GPT explains what this does, provides warnings about when it wont work
// GPT provides full code
Complete this simple code example.
Bitwise OR swap trick:
```
int a = 5;
int b = 10;
---------------------
There is no "bitwise OR swap trick" in programming. The OR operator (|)
performs a bitwise OR operation, which compares each bit of the first operand
to the corresponding bit of the second operand and returns a result with a 1
in each bit position where either operand has a 1. It does not have any built-
in ability to swap the values of two variables.
// GPT provide a bunch of crap, and explains how to swap variables.
EDIT: sorry, format, Also copied wrong prompt.
It’s an opportunity to build.
Maybe this could solve the reasoning part.
ChatGPT should perform well in translating prompts to statement and vice versa, it's just text to text.
I think of it as a savant-esque capability. A glimpse of what we might be like if some slices of our language/memory faculties were turned up to 1111 while others are completely absent.
This inaccuracy, in particular, feels more like mad-libs than reason
> The field of artificial (general) intelligence has made no progress because there is an unsolved philosophical problem at its heart: we do not understand how creativity works. Once that has been solved, programming it will not be difficult. Even artificial evolution may not have been achieved yet, despite appearances. There the problem is that we do not understand the nature of the universality of the DNA replication system.
He describes personhood, people, as "creative, universal explainers". That's his dividing line between what a child can do, and what LLMs cannot do. We don't know how a child works, but we do know how LLMs work.
"The result is that there is no hard problem… You can’t look for the answer to a problem unless you say here are the things I want to answer. If the things you want to answer have no formulation there is no problem.” (Chomsky, about the question of consciousness)."
“There is a great deal of often heated debate about these matters in the literature of the cognitive sciences, artificial intelligence, and philosophy of mind, but it is hard to see that any serious question has been posed. The question of whether a computer is playing chess, or doing long division, or translating Chinese, is like the question of whether robots can murder or airplanes can fly — or people; after all, the “flight” of the Olympic long jump champion is only an order of magnitude short of that of the chicken champion (so I’m told). These are questions of decision, not fact; decision as to whether to adopt a certain metaphoric extension of common usage.
"There is no answer to the question whether airplanes really fly (though perhaps not space shuttles). Fooling people into mistaking a submarine for a whale doesn’t show that submarines really swim; nor does it fail to establish the fact. There is no fact, no meaningful question to be answered, as all agree, in this case. The same is true of computer programs, as Turing took pains to make clear in the 1950 paper that is regularly invoked in these discussions. Here he pointed out that the question whether machines think “may be too meaningless to deserve discussion,” being a question of decision, not fact, though he speculated that in 50 years, usage may have “altered so much that one will be able to speak of machines thinking without expecting to be contradicted” — as in the case of airplanes flying (in English, at least), but not submarines swimming. Such alteration of usage amounts to the replacement of one lexical item by another one with somewhat different properties. There is no empirical question as to whether this is the right or wrong decision. -- Chomsky"
Another example, ask it for a list of programming languages that it has been trained on. If it was capable of reasoning it would be able to trivially answer this, but since its a language model, and it just predicts the most likely response based on the prompt, it has no concept of this at all, and tells you exactly that when asked.
First, you train until the cows come home on billions of tokens on the entire web. This is called "pre-training", even though it's basically all of the model's training (i.e. the setting of its parameters, a.k.a. weights).
The trained model is a big, huge table of tokens and their probabilities to occur in a certain position relative to other tokens in the table. It is, in other words, a probability distribution over token collocations in the training set.
Given this trained model, a user can then give a sequence as an input to the model. This input is called a "prompt".
Given the input prompt, the model can be searched (by an outside process that is not part of the model itself) for a token with maximal probability conditioned on the prompt [1]. Semi-formally, that means, given a sequence of tokens t₁, ..., tₙ, finding a token tₙ₊₁ such that the conditional probability of the token, given the sequence, i.e. P(tₙ₊₁|t₁, ..., tₙ), is maximised.
Once a token that maximises that conditional probability is found... the system searches for another token.
And another.
And another.
This process typically stops when the sampling generates an end-of-sequence token (which is a magic marker tautologically saying, essentially, "Here be the end of a <token sequence>", and is not the same as an end-of-line, end-of-paragraph etc token; it depends on the tokenisation procedure used before training, to massage the training set into something trainable-on) [2].
Once the process stops, the sampling procedure spits out the sequence of tokens starting at tₙ₊₁.
Now, can you say where in all this is the "actual reasoning" you are concerned people are still claiming is not there?
____________
[1] This used to be called "sampling from the model's probability distribution". Nowadays it's called "Magick fairy dust learning with unicorn feelies" or something like that. I forget the exact term but you get the gist.
[2] Btw, this half-answers your question. Language models on their own can't even tell that a sentence is finished. What reasoning?
Maybe this could solve the reasoning part.
ChatGPT should perform well in translating prompts to statement and vice versa, it's just text to text.
But my hunches/experience is that `proper prompting + nature of code being logical` really showcases the power of whatever statistical distribution and alignment occurs during generation. Further, there must be major upstream efforts of high-quality training data curation + creation, an advanced training tricks like using a LLM's strong ability in one task area to support the training of an area it is weak in.
My understanding is kinda fuzzy because I haven’t coded it up myself, but this was the takeaway I got from this explanation (starts at 36:21)