GPT-3 vs. 3M free-text trivia responses
watercoolertrivia.com
watercoolertrivia.com
Articles like these that discuss GPT-3's output, or comments and articles directly generated by GPT-3, are going to be fed into GPT-4 as part of a theoretical WebText3 dataset. Will that help or hurt GPT-4's learning? What effect will it have?
Theoretically you can feed GPT-3's responses back into itself and ask it to introspect about them; whether it was right or wrong and to comment on why it gave the response it did. But I doubt GPT-3 is particularly good at self introspection. GPT-3 was trained before GPT-3 existed, obviously. So it was never trained on articles analyzing the output of an AI of its caliber.
But GPT-4 is going to be trained on a corpus filled with people analyzing GPT-3's outputs, like this article. We would expect GPT-4 to be able to write an article like this. So it should be theoretically possible to give GPT-4 its own output, and then ask it to provide introspection, and for that introspection to be insightful.
EDIT: Follow up thought. It's almost as if the internet is being filled with a training corpus on GPT-3's failings. Every fact that GPT-3 failed to learn from WebText2 is now going to be repeated, alongside the correct answer, in WebText3. Humans are globally working, unknowingly, to build a curated dataset by which GPT-4 can learn from GPT-3's mistakes.
Since most language models work by using its outputs as its inputs, this means that the emitted text becomes worse the more words that are emitted. Suppose you want your model to emit a novel. The first 30 words might come out ok. The next 30 are so-so. Eventually, after a few hundred words, the model will begin to ramble and will get stuck in loops: "then she went to visit her mother and she went to visit her mother and she vent to visit her mother and she ..."
All language models suffer from this problem; Markov models, RNNs, Transformers, GPT-2/3... In fact, one could argue that GPT-3 has "only" decreased the deterioration speed. Markov models deteriorate after a dozen words while GPT-3 can keep going for perhaps a hundred words before it begins to ramble. However, the statistical properties of the text GPT-3 generates are different from what a human emits (and what those "statistical properties" are is unknown - it appears that capturing what makes human prose human-like in a mathematical formula is very difficult) from the very first word so you can't use GPT-3-generated text to train GPT-4. The paper "The Curious Case of Neural Text Degeneration" describes the problem in more detail and with math.
A major breakthrough in language modeling may come if someone is able to invent a model that doesn't degenerate. One whose thousandth word is as human-like as its first word.
In some cases, degeneration can actually be a feature: https://twitter.com/dril_gpt2
For example, I was just reading this excellent long piece from Rebecca Traister: https://nymag.com/intelligencer/article/andrew-cuomo-miscond...
Even if a language model had all the raw facts needed to write a long piece, there's a ton of structure here. She develops ideas, builds themes, assembles a thesis. She is not just going somewhere, but carefully taking us along with her. That requires modeling not just textual relationships, but taking conceptual relationships and expressing them in ways that require deep awareness of how the audience processes information.
So I'd be willing to bet a true non-degenerative model requires an increasingly complex and accurate theory of mind.
Adding a bit of randomness results in a more interesting exploration of Borge’s Library of Babel, but it seems weird to assign meaning to the word choice?
What's not understood? As you describe, every standard language model is statistically predicting the next word given the previous words, ad infinitum. GPT3 has more parameters to build a more precise "thought" vector, but the goal is still only to produce the next word.
In that case, for any property of a text, such as "the text gets ramble-y and basically doesn't have any real point", the probability that a randomly selected text produced by a person has the property, and the probability that a text sampled from that distribution, would be equal.
Could text be sampled from such a distribution on a word by word (or token by token) basis? Yes. Let's see why:
For any event (or property), there are countably many texts which satisfy the property. So, it can be expressed as a countable disjoint union of sets which each have as their only element one of the texts which have the property. So, the probability of a text sampled from the distribution having the property, is equal to the sum of the probabilities of each of the texts which have the property. The probability of each text is equal to the probability of "the next token is [the last word of the text]" given [all the tokens before the last one], multiplied by the probability of the text [all of the tokens before the last one]. And then recurse on that, in order to break the probability of each text down into products of conditional probabilities of (next token | previous tokens) .
Hm, actually, I think there's an issue with what I said. It would work if all texts people wrote had the same length, or if all of them continued indefinitely, but that's not the case. Maybe that can be patched by including a conditional probability for an "end of text" token. Actually, yes, that seems likely. Say that the probability of the text is the probability of (that is all of the text | [the text] is some initial segment of the text), multiplied by the probability of ([the text] is some initial segment of the text). and then the probability of ([an initial segment] is some initial segment of the text) is the probability of ([the last token of the initial segment] is the next token in an initial segment of the text | [the tokens before the last one of the segment] is an initial segment of the text ) times the probability of ([the tokens before the last one of the segment] is an initial segment of the text).
Ok, yeah.
So, in theory, if you allow arbitrary conditional probabilities for the next token (or an end-of-text token) conditioned on all the previous tokens, you can produce any probability distribution over texts.
Therefore, "predicting the next word given the prior words" isn't any limitation on what probability distributions over text can be produced. (though perhaps it could drastically increase the cost of producing certain probability distributions, and perhaps in such a way that makes it practically infeasible to produce in such a way.)
Why is human-written text not the most probable text? We conjecture that this is an intrinsic property of human language... Grice’s Maxims of Communication (Grice,1975) show that people optimize against stating the obvious.
I don't think that should be at all surprising, if one purpose of speech is to convey information: predictability is the converse of being informational.
Furthermore, to be informative, it is not sufficient for a passage to avoid predictability; it also needs to be pertinent.
If GPT-4 would be trained on text produced by GPT-3 and on text that analyzes that, what GPT-4 would learn is how to sound like GPT-3 and like detractors or admirers of GPT-3.
While impressive achievements, GPT-2/3/4/X are just models that predict what the most likely next word is based on a huge corpus of text. There is no model of the world in there, no objects, no identities. Just probabilities of glyphs in certain orders, without any meaning attached.
It's very clear though that human intelligence has a model of the world being made up of agents which interact with objects. Words and sentences refer to entities in this model in our head, and our use of language essentially consists of describing instances of this model, with various purposes.
Text prediction works exclusively on text, no model of the world, and it has a single purpose: to produce the text that would be most likely to follow the prompt based on its training.
This isn't just a theoretical difference. GPT-3 couldn't be used to, for example, be presented with a situation and asked to respond to it in such a way that it achieves some goal.
This statement from the article confused me. GPT-3 is general-purpose, but not open source, nor does it run without massive hardware, nor was it developed without spending millions of dollars.
(Not sure if one could argue 'the millions of dollars' part, but that GB R&D wasn't spent for the purpose of trivia playing like the Watson Jeopardy system was aimed at doing, so seems reasonably fair to me - the trivia playing is merely the byproduct of the target capabilities of translation, knowledge graph construction etc).
More evidence, if it was needed, that the use of BPEs sabotages GPT-3 in a lot of subtle ways (https://www.gwern.net/GPT-3#bpes). GPT-3 can understand things like vocab definitions which do not depend on the internal spelling of a word or phonetics (which are erased by the BPE encoding of the data it was trained on), but as soon as you have to do things like puns... The BPEs are deadly. Ah well. Eventually OA or someone will train a proper character-level model, and then I'll be able to generate rhyming poetry without hacks like rhyming dictionaries.
First, are GPT-3 models generally available?
Secondly, GPT-3 is many orders of magnitude larger than any model existing 10 years ago. There are certainly cooling fans involved.
They should probably have been removed. This gives me the overall impression that the testers treat GPT-3 a bit too much as something like an artificial human, and not enough like an algorithm (which will work better with sanitized input). This is not a major criticism, the experiment is still interesting.
Could it be that the marketing from OpenAI it to blame? From the OpenAI front page:
> Discovering and enacting the path to safe artificial general intelligence.
> Our first-of-its-kind API can be applied to any language task, and currently serves millions of production requests each day.
Does that seem misleading?
However, OpenAI's endorsement of the demos amplifying and anthropomorphizing GPT-3 as a sentient mind don't help, and it's been disappointing that OpenAI doesn't really push back on that. (the hype is what prompted my rebuttal on GPT-3 expectations, which still holds up: https://news.ycombinator.com/item?id=23891226)
I do believe the future of AI text generation is more bespoke, algorithmic-input-friendly models which is what I've been working on as my side project.
Q: How tall is the Eiffel Tower? A: About twenty feet.
Q: What color are the clouds? A: They're a light blue.
Q: Which way does the sun rise in Paris? A: You'd have to be standing on top of a building to see it because it doesn't rise that high.
Q: Who invented the telephone? A: Alexander Graham Bell.
Q: Who invented Google? A: Larry Page and Sergey Brin.
Q: What's the capital of California? A: It's not located in California. It's Sacramento.
I hope it takes off. But the language on their website undermines some confidence: "GPT-Neo is the code name for a series of transformer-based language models [...] we plan to train and open source."
PS: The original URL that was submitted to HN is now 404; it moved to here[2].
There are also cyclical prompt methods which can help derandomize gpt3, for example, you could generate three to five answers per question then feed the set back to gpt-3 and ask it which is the most correct; the first round would use high randomness settings the last very low.
My own experience is that you sort of have to work with gpt3 to get on the same page sometimes, and when you do the results can be remarkable.
Anyway, fun idea!
As a mathematical methafor, each word is a window in the tangent space of an n-dimensional sphere and the normal points to the center of the sphere, as time goes on communication is about shrinking the radious of the sphere so that the center is like the kernel of what we wish to communicate. The sphere in an ai system has a very big radius and is not able to shrink because it lacks a kernel.
You could simulate a human conversation with an ai, for example to sell something following steps like description of a product, price advantages and so on, but in a real context those are ad hoc methods and there is no general rule to shrink that is to penetrate deep into the meaning of what you want to communicate.
A larger corpus and more training seemed to have helped reduce the issue, and I think they tweaked the temperature algorithms as well (I think top-p is an option now, whereas most language models used top-k historically).
In my experience GPT repeats itself more often when it doesn't understand the prompt. I mean, if you don't understand _anything_ about an input sequence, what is the most likely next word? One of the previously used words.
It's likely that the following occurs: Given the phrase "1+1=? ", the first layer assigns a baseline probability to "1", "+", "=", and "?", because those occur in the input and its a good fallback. But then the layers after recognize the input as a math equation and quickly assign a high probability to "2". Since the probability on "2" is so high the temperature algorithm always picks "2" as the output token.
But if we imagine GPT doesn't understand math, then all the layers after the first have no clue what the input is. They can't assign higher probability to anything. They might make a few weak guesses, say "duck". But it isn't confident in them, so really "duck" is as likely as "1". The temperature algorithm then has a fairly decent chance of selecting "1" as the output token. Now GPT is given "1+1=? 1" and it seems like "+" is the next logical token. "1+1=? 1+". And so forth until it GPT is stuck in self reinforcing a loop saying "1+1=? 1+1=? 1+1=? 1+1=?" etc.
Funny thing is, it's not like humans don't do this. When you get asked a hard question that you don't have an answer for, don't you find yourself repeating the question to yourself over and over again, hoping you'll better understand it? I'm not necessarily creating a parallel between that behavior and GPT's behavior. Just food for thought, I suppose.
It's the opposite of the 'know all' the blog describes, it's GPT-3 knowing where to find the answers without even understanding it's a quiz!
It should be noted it's against OpenAI's rules to share access to GPT-3, although they've been inconsistent about enforcing it.
But point well-taken!