GPT-3 can run code
mayt.substack.com
mayt.substack.com
Regarding GPT-3's "guesstimates," intuitively it feels like the network has to guess because it hasn't been given a way to do exact computation--a neural network is built out of nonlinear functions--even if it "understands" the prompt (for whatever value you want to give to "understand").
Are there any techniques that involve giving the model access to an oracle and allowing it to control it? To continue the analogy, this would be the equivalent of giving GPT-3 a desk calculator.
If this is a thing, I have other questions. How do you train against it? Would the oracle have to be differentiable? (There are multiple ways to operate a desk calculator to evaluate the same expression.) Also, what control interface would the model need so that it can learn to use the oracle? (Would GPT-3 emit a sequence of 1-hot vectors that represent functions to do, and would the calculator have "registers" that can be fed directly from the input text? Some way of indirectly referring to operands so the model doesn't have to lossily handle them.)
In the Retrieval-Enhanced Transformer (RETRO) paper a large language model was coupled with a similarity based text index. It can populate the prompt with relevant information from the index thus being more grounded and update-able.
In another paper (AlphaCode) the language model was coupled with a compiler and could run programs and check if they match the expected outputs for a few test cases. The model was able to solve competition style coding problems above average human score.
In another paper (Language Models as Zero Shot Planners) a language model generates commands to navigate a virtual home environment and performs tasks. The knowledge in the LM helps in quickly learning tasks.
A recent one can learn new concepts by simple conversation, then apply them where necessary. You can talk-train your model. (Memory assisted prompt editing to improve GPT 3 after deployment)
So the trend is to add "toys" on language models - a simulator, a compiler, a search engine, a long term memory module.
I'd like to see a recursive language model, that can sub-call itself to decompose problems.
Anyway, today another great paper dropped on self-distillation: "STaR: Bootstrapping Reasoning With Reasoning" https://arxiv.org/abs/2203.14465 , Zelikman et al 2022.
I tried a very simple and specific version of this a few years ago (Recursive Application of Recurrent Neural Networks) and it worked great for intent parsing: https://github.com/spro/RARNN
Would like to see what "real" researchers with more modern models could do with the concept.
I am not sure if I am thinking of the right study, but as far as I remember the model included a human wading through and filtering solutions and while there may have been a compiler attached they also scored themselves. The marketing blurb of course tried to make it sound as if they had competed.
Here's a good analysis of the paper: https://www.youtube.com/watch?v=s9UAOmyah1A
Which model? Sauce please
Is there a way to allow models to say "let me think about this some more"? With language models like GPT-3 you emit one token per inference iteration, with its previous output fed back in as input/state. Can models opt out of providing a token, but still update state? That would allow it to break up the computation into discrete steps.
RNN outputs "confidence" bit which can guide computation to perform more steps to obtain more confidence in the result. Essentially, RNN asks "let me think about that some more".
But, separate ablation study found that if you just drop confidence bit altogether and allow RNN to compute some more every time (e.g., always perform 4 computations on single input for 1 output), you get same or better results without extra complexity of training.
There is also Microsoft Research's paper I can't find right now about the variable computation for image classification where there is a "confidence" bit at some of the final layers - if lower layer is cinfident enough, it's output will be used for classification, otherwise the output of that layer will be passed into more transformation of upper layers.
Do they saw what happens if you do both? Perhaps the “benefit from a higher computation/per cycle” phenomena and the “benefit from signalling relative computation resource allocation” one are different.
I guess I’ll have to try and read the paper, but I’m new to the literature and am clueless about the current state of research.
It would of course have to be penalized in some way for [THINKMORE]ing to avoid infinite processing time. It would have to learn to reason at what point diminishing returns would kick in from continuing to [THINKMORE] VS recording its best answer. The penilization function would have to take into account remaining tokens that would fit in the transformer prompt.
Prompt: "Question: Solve three plus six.
Answer:
a=3
b=6
a+b
Question: Solve twelve times fifteen. Answer: a="
And the model dutifully answered:
"a=12
b=15
a*b"
Which you could feed directly to a python console.
This kind of approach, where you make a long prompt to make the model understand the kind of result you want is named "prompt engineering" and I find it crazy how close we get to robopsychology.
# Instruction def f(x): if x > 30: return "too large" else: return x + 3
How the hell it is different from the programmer writing the python function herself and where exactly is the "intelligence" in this?
There are two issues here. One is the lack of working memory, which means that there is very little scratch space for calculating things with a meaningful sequential depth. GPT-3 is very unlike traditional evaluation methods in this regard, in that it is easier for it to interpret the meaning of a program you give it and then intuit the result given the context than it is to mechanically execute its steps.
The other issue is the text encoding, which makes it much harder for GPT-3 to do digit-by-digit operations. Many arbitrary numbers are just their own token. A fixed length number to us looks like a fixed number of characters, but for GPT-3 they can be and almost arbitrary number of tokens divided into almost arbitrary chunks. Using thousands separators is very helpful for it.
If you account for these and design a prompt that mitigates them you can get much stronger results. Here is an example: https://news.ycombinator.com/item?id=30299360#30309302. I managed an accuracy of 42% for 3-by-3 digit multiplication.
It's a language model. It can generate text, not "calculate things".
If you give it the right prompt, it will generate the right text, but if there's any computation going on, that's you computing the right prompt.
See Clever Hans:
Oh, I think I see what you mean. Thank you for clarifying. So, no, I didn't mean that the prompt is engineered to make it look like the model is performing a calculation. I meant that GPT-3 has memorised instances of arithmetic operations and in order to retrieve them from its memory the human user must figure out the right prompt. I wrote "that's you computing the prompt", not "that's you computing the result".
The prompt is like a SQL query, right? If you don't enter the right query, you don't get the right results. That's the point of all those people on the internets fiddling with their prompts- it's like they're trying to query a database, but they don't know what the right syntax is for their query, so they tweak it until it returns the results they want.
For example, the OP mentioned thousands separators being very helpful to the model. That's because it's memorised more arithmetic results with thousands separators, than without. So you're more likely to get the right results out of it if you use thousands separators.
Also because like the OP says GPT-3 has a separate concept for a digit and a string of digits and a separate one again for a string of digits and other symbols. "9999" is, in its model, a different thing than "9,999".
Which, btw, is why it can't calculate. Because to calculate, a system must have a representation of the concept of a number. Otherwise, calculate- with what?
This distinction is not that clear though. If you can predict well the output of a function, that's equivalent to executing the code.
Still, technically you're not executing the code.
I agree that it's fundamentally different, but I'm not exactly sure how, and I think it's subtler than you're suggesting.
We call computers deterministic despite the fact that they don't with perfect reliability perform the calculations we set them. The probability that they'll be correct is very high, but it's not 1. So the requirement we have for something to be considered deterministic is certainly not "perfectly a hundred percent of the time", as the parent to my comment suggested.
It's a non-deterministic algorithm, of which many kinds exist. Producing different answers that are close-ish to correct is in fact what a Monte Carlo algorithm does. Not that you'd use GPT3 as a Monte Carlo algorithm though, but it's not that different.
Close-ish to correct makes sense for some problems and makes no sense at all for others.
Imagine a C compiler that does aggressive optimizations - sacrificing huge amounts of memory for speed. On one hand, it even reduces computational complexity, on the other it produces incorrect results for many cases.
GPT-3 as presented here would be comparable to that. Neither are equivalent to executing the original code.
Meanwhile, the result of something like gcc is, even if it runs on a computer with faulty RAM.
Speed and memory is orthogonal to my point, which is about the output of two methods of arriving at an answer. I'm obviously not saying GPT-3 is anything like as efficient as running a small function.
What distinction are you drawing between the output of an interpreted program and a compiled program?
My take is not that GPT-3 can run code, but rather that GPT-3 has memorized what code looks like and what the output looks like.
Proof:
[GPT-3's "proof"]
Let p be a prime number. Then by definition, p is a positive integer that has no positive divisors other than 1 and p.
Now consider the integer q = p + 1. We will show that q is not a prime number, which will prove that there are infinitely many prime numbers.
Since p is a positive divisor of q, q cannot be prime.
[...another attempt with the same prompt]
We know that there are infinitely many natural numbers. Let's suppose that there are only finitely many prime numbers. Then we can list them all:
p_1, p_2, p_3, ..., p_n
Now, consider the number
N = p_1 p_2 p_3 ... p_n + 1
This number is greater than any of the prime numbers in our list, so it cannot be divisible by any of them. But it is also a natural number, so it must be prime. This contradicts our assumption that there are only finitely many prime numbers, so we must conclude that there are infinitely many prime numbers.
~~Personal remark: I had to give it the same prompt like 5 times to get a correct-ish proof. The first attempts came out like the first example.~~ The proof is still wrong.
- It's a proof by contradiction - The key step is in taking the finite list of primes, multiplying them together, and adding 1
I then try to flesh out the details, it might take a second to realize that this new number is also prime, and then a few moments more to remember the exact rationale why.
Along the way the proof lives in a kind of superposition where I'm not clear on the exact details. The "proofs" you gave here seem to be serializations of a similar superposition! GPT-3 seems to remember the proof about as well as I do, but it's missing the final sanity check which tweaks the proof until all the pieces correctly fit together.
In this case, you seem to be performing a version of this sanity check by running the prompt multiple times until a correct answer comes out. I wonder if it's possible to prove something more obscure using a similar process: GPT-3 comes up with ideas and the human sanity checks.
Not necessarily, it might be composite, but in this case one of it's prime factors will necessarily not lie in the supposed list of primes, therefore also a contradiction.
The first counter example to "If L := {P0,P1,..,Pn} is a list of primes, then prod(L)+1 is prime" is {2,3,5,7,11,13}, their product is 30030, and 30031 is a composite of 2 primes, none of which are in the list.
For instance, 2 * 3 * 5 * 7 * 11 * 13 + 1 = 30031 = 59 * 509.
Assume p1, ..., pn is a finite list of primes. The sum p1+...+pn+1 is divisible by a prime, because every natural number> 1 is. However, it's not divisible by p1,...,pn, hence there must be an additional prime not in the list.
(I think you're right though that GP's "contradiction" doesn't work)
Never thought of using "by definition, all numbers can be divided by a prime" tu merge the two cases. It's not that shorter, but is IMHO quite elegant, I'll remember it. Thanks for correcting me.
Euclid's original proof of the theorem is of the form "for any list of primes, I can find an additional prime" [0], and for good reason: in Ancient Greece, thinking of infinity, or infinite sets, as a concrete object that you could manipulate would have seemed weird.
But the proof variant where you produce a contradiction doesn't really get into the set-theoretic details either. All it does is say: "Assume there is a finite list of all primes. Derive a contradiction. Therefore there is no such list." That's pretty much equivalent to the direct proof, it's just using different logical inference rules.
[0]: http://aleph0.clarku.edu/~djoyce/java/elements/bookIX/propIX...
These answers are the sort one might expect from something that has a vast memory for what it has seen before, and an ability to draw huge networks of syntax-level associations and generalizations from all that text, but is not so strong on semantic associations and generalizations that are not manifest at the level of syntax. What surprises me is how successful that has been.
[0]: Self-Consistency Improves Chain of Thought Reasoning in Language Models https://arxiv.org/abs/2203.11171
I guess it would stitch together some more seemingly sensible statements that also don’t quite add up to a rigorous proof?
Does it actually "run" the code? Like, if it was looping over 1 billion iterations would it take 1B times longer than if it was just one iteration? I have so many questions.
If you read through all of the internet once, would you know that range() is zero indexed?
> Like, if it was looping over 1 billion iterations would it take 1B times longer than if it was just one iteration?
It clearly cannot, because querying the network for a token executes the exact same sequence of operations every time.
But it's very impressive that it can basically recognize the Collatz Conjecture in the code and mostly guess in the right ballpark for the results.
The fact that it's just liking (in a loose sense) inputs to inputs it has seen is quite visible in the f(g(x)) vs g(f(x)) behavior - the former is significantly more common, so it struggles to work with the latter.
>>> for i in range(3): print(i)
0 1 2
I don't know if this example helps, but a computer can generate (pseudo-)random numbers by executing an algorithm. A pair of dice can also generate random numbers because they're thrown so that they land at random and someone has marked pips on their faces that a human can read as numbers. The result may be similar, but one is generated by a computation and the other by a random process. The random process is not a computation. It's a random process.
(Or just unpredictable).
Being able to intuit what a block of code does is actually a core skill; having to actually step through code in your head is slow and difficult.
Let's hope it doesn't turn into something like SCP-079...
This is an interesting benchmark because it is a very difficult problem, however: GTP has both everything it needs to do this without needing a fundamental improvement to the core of GTP (this process is more of a science than art) and using automated UI testing GTP can check if its solution worked.
Thus this challenge is in the realm of what GTP already is, however, once it can do this it will have massive implications for how software is built.
It's hard enough for people to faithfully port an application. People who participate and live in the world that makes up our reality. Leaving this up to an AI will at best flood us with low quality junk. At worst it's actively harmful.
But it can't because it doesn't have the right structure (e.g. GPT-3 finishes in a finite time, a program in a real programming doesn't necessarily!)
GPT-3's greatest accomplishment is that it has "neurotypical privilege", that is if it gets an answer that is 25% or 95% correct people give it credit for the whole thing. People see a spark of intelligence in it the way that people see faces in leaf axels or in martian rock formations or how G.W. Bush looked in Vladimir Putin's eyes and said he got a sense of Putin's soul. (That was about the only thing in his presidency that he later said he regretted!)
As an awkward person I am envious because sometimes it seems I get an answer 98% correct or 99.8% correct and get no credit at all.
Proof is, that indeed humans do get the wrong answer in quizzes like these sometimes!
So i cannot understand this point of view of diminishing it as "spark of intelligence". It is exactly what advertised: a very big step forward towards real AI, even if definitely not the last one?
GPT-3 gets the wrong answer because it has memorised answers and it generates variations of what it has memorised. It generates variations by sampling at random from a probability distribution over what it's memorised. If it has the correct answer memorised, sometimes it will generate the correct answer, sometimes it will generate a slight variation of it, sometimes it will generate a large variation of it, sometimes it will generate something completely irrelevant (i.e. with a very small probability).
Failure is not an exclusive characteristic of humans. In particular, any mechanical device will fail, eventually. For example, a flashlight will stop functioning when it runs out of battery. But not because it is somehow like a human and it just got it wrong that one time.
It has the special talent of hijacking your own intelligence to make you think it is intelligent.
People understood this about the 1966 ELIZA program but intellectual standards have dropped greatly since then.
Well, even with those small numbers, it's wrong. The first "2" after the dot should not be there. The result it gives is 16/7, not 20/7.
See also the "x = x + x three times", for which the result is not random but the result for the same thing... Two times instead of three (so result/2). That heavily smells like it has read sites that had nearly the same code on them.
https://beta.openai.com/playground/p/o4qZWSXVz8JMmVaI9j9NMIK...
Alternatively you could just scale 'till the problem solves itself.
It's obviously not. To handle infinite loops it needs to solve the halting problem. Which is not possible.
print("A");
potentiallyHalts();
print("B");I mean, in the strictest sense that isn’t Turing complete either, because when you have a timeout you cannot run every program a theoretical Turing machine could. But then no practical computer is, because resources are always constrained (e.g. finite memory instead of an infinite tape). So when we talk about something being Turing complete, we usually disregard the resource limitations and effectively substitute something like “we mean Turing complete in the sense that it would be if we also had infinite memory and time”.
So, I still don’t understand why GPT-3 would have to (impossibly) solve the halting problem to be Turing complete[1], but everything else including a python interpreter or lambda calculus doesn’t.
[1] Note that I don’t assert that GPT-3 could or could not be Turing complete, I just don’t know why the halting problem predicates that.
As for loops: a Turing-machine could do infinitely many loops, so Turing-completeness implies that a system can do the same. If GPT-3 can't do infinitely many loops, it's not strictly Turing-complete; and if it can't do many loops, then it wouldn't seem like a meaningful approximation of a Turing-complete system.
(see p. 21) https://arxiv.org/pdf/2203.07814v1.pdf
It is being discussed because it is surprising that GPT-3 can do it at all. It is worth investigating what types of emergent knowledge and behavior are encoded in the trained network, as the boundaries of its capabilities may help illuminate future neural network architecture design.