Can GPT-4 and GPT-3.5 play Wordle?
twitter.com
twitter.com
I'm just completely baffled how anything in the training procedure could allow the LLM to learn information about the structure of tokens. Does the tokenization process not treat every token (which I thought usually maps to a word) as an "opaque blob"?
It's not perfect, though. There are funny failure modes where you can ask the model things like "does the word foobar have a 'b' in it" and it will say "no". (It doesn't fail in the 'foobar' case, but it does fail for some of these cases -- this was just an illustration).
Me: I would like to play a would game with you. I'm going to tell you a made-up word, and then ask you a question about it. Ready?
ChatGPT: Sure, I'm ready to play! Let's get started.
Me: Does the made-up word plukatoon have the letter e in it?
ChatGPT: No, the made-up word "plukatoon" does not have the letter "e" in it.
-- Note, at this point I was really surprised it got it right. I chose "e" thinking that it would default to yes given the popularity of e.
Me: Correct! Does the made-up work ziddlebum have the letter e in it?
ChatGPT: No, the made-up word "ziddlebum" also does not have the letter "e" in it.
-- OK, proof that it's "not perfect" as you point out
Me: Does the made-up word folladel have the letter o in it?
ChatGPT: Yes, the made-up word "folladel" does have the letter "o" in it.
-- Edit, and finally:
Me: How do you spell the made up word "platoggle"?
ChatGPT: The made-up word "platoggle" is spelled as follows: P-L-A-T-O-G-G-L-E.
OK, so that last one TOTALLY blew my mind. I'm guessing it's never seen the word "platoggle" before, so even if it learned previous spelling associations with other tokens, it couldn't possibly understand how to spell "platoggle" if it's treating it just as a single, never-before-seen, opaque token, could it?
The more I play with this stuff the more existential crises I have.
That's not how the tokenizer works. A novel word like "platoggle" is decomposed into three separate tokens, "pl", "at", and "oggle". You can see for yourself how prompts are tokenized: https://platform.openai.com/tokenizer
If that is a inference time performance or training time performance or a model size issue or just total nonsense, I wouldn't know.
So notice it doesn't contain any vowels, since almost all consonant-vowel pairs are sufficiently frequent in the training text as to be tokenised at least as a pair. E.g. "guq" is tokenised as "gu" + "q", since "gu" is common enough.
(Compare "gun" which is just tokenised as a single token "gun", as it's common enough in the training set as a word on its own, so it doesn't need to tokenise it as "gu"+"n".)
The only exceptions I found with consonant-vowel pairs being tokenised as pairs were ones like "qe", tokenised as "q" + "e". Or "qo" as "q"+"o". Which I guess makes sense, given these will be low-frequency pairings in the training text -- compare "qu" just tokenised as "qu".
(Though I didn't test all consonant-vowel pairs, so there may be more).
Spell the longest state name backwards, omitting one t of your choice
> The longest state name is "Massachusetts." Omitting one "t" and spelling it backwards, we get "sttesuhcassaM."
I wouldn't have it quit it's day job yet.
It's not great at random substring manipulation, but spelling words is so common it almost certainly has internalized some structure akin to a programmatic representation of decomposing a string into characters.
The biggest difference I see is that it doesn’t have consciousness: the capacity to detach it’s simulation of the world of the present.
When we are conscious we jump in a simulation in our head we imagine a future where we do one thing and then we get one kind of result. If we don’t like the results we have in our simulation we jump to an alternative and so on until we find a path that we like.
We can attach ours simulation to past memories and again simulate alternative, to try to integrate what happened to us. GPT doesn’t seem to have that yet, it’s always in the present moment, like a dog, no worry for the future.
As to how it ends up this way, that's an interesting question, but it kinda makes sense in a broad way that if you try to "lossily compress" a lot of interrelated data, at some point the only way to continue doing so is by inferring the underlying concepts - i.e. building an internal model of whatever the inputs describe. And, notably, we had to make the models large enough before results indicative of that manifested, regardless of the amount of text thrown at it during training.
1) An English dictionary as input.
2) List of words that start with "app" wiki page as input.
3) Other alphabetically sorted pieces of text.
4) Elementary school homeworks for spelling.
5) Papers on glyphs, diphthongs, and other phonetic concepts.
You begin to recognize that the tokens in these lists appear near each other in this strange context. You hardly ever see token 11346 ("apple") and token 99015 ("appli") this close to each other before. But you see it frequently enough that you decide to nudge these two tokens' embeddings closer to one another.
Your ability to predict the next token in a sequence has improved. You have no idea why these two tokens are close every ten millionth training example. Your word embeddings start to encode spelling information. Your word embeddings start to encode handwriting information. Your word embeddings start to encode phonic information. You've never seen or heard the actual word, "apple". But, after enough training, your embeddings contain enough information so that if you're asked, ["How do", "you", "spell", "apple"], you are confident as you proclaim ["a", "p", "p", "l", "e", "."] as the obvious answer.
Is it possible for the current generation of LLMs to assign confidence intervals to their responses?
That's my main qualm with ChatGPT so far: sometimes it will give you an answer, but it will be confidently incorrect.
Take this with a grain of salt though, I'm far from an expert, and it's been a while since I've played around with that feature.
> GPT-4 can also be confidently wrong in its predictions, not taking care to double-check work when it’s likely to make a mistake. Interestingly, the pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, after the post-training process, the calibration is reduced (Figure 8).
pages 10-11: https://cdn.openai.com/papers/gpt-4.pdf
When I read the GP description referring to "embedding" above I thought of the perceptron.
Definitely not supernatural at all. The act of making an automaton that "can perceive" feels to me like it's closer to the opposite. Taking that which might seem mystical and breaking it down into something predictable and reproducible.
Of course, those coordinates are not the only way in which the object can be represented, but for a certain problem context, these location coordinates are useful.
Given objects A,B,C, or rather, given their coordinates, one can tell which two are closest to each other, or you can find the point D that is the other point of the parallelogram ... this. In fact, it allows you to do similarity tests like "A:B :: C:D". This is through standard vector algebra.
Now, imagine each word associated with a 100-dimensional vector. You can do the same thing. Amazingly, one can do things like "man:woman ::king: ...." and get the answer "queen", just by treating each word as a vector, and looking up the inverse mapping for vector to word. It almost feels ... intelligent!
This embedding -- each word associated with an n-D vector -- is obtained while training neural nets. In fact, now you have readymade, pre-trained embedding approaches like Word2Vec.
During training, each token (or word) gets an Embedding assigned.
Critically, _similar words will get similar embeddings_. And "similar words" could mean both semantically or (as was the example) syntactically ("apple" and "appli").
And being vectors, you can do operations on them. To give the classic example, you could do: Embedding(`king`) + Embedding(`female`) = Embedding(`queen`).
basketball ==> (1.0, 0.7) # heavier, redder baseball ==> (0.2, 0.2) # less heavy, less red
When an LLM (large language model) is fed a word, it transforms that word into a vector in n-dimensional space. For example:
basketball -> [0.5, 0.3, 0.6, ... , 0.9] # Here the embedding is many, many numbers
It does this because computers process numbers not words. These numbers all represent some property of the word/concept basketball in a way that makes sense to the model. It learns to do this during it's training, and the humans that train these models can only guess what the embedding mappings it's learning actually represent. This is the first step of what a LLM does when it processes text.
This is a high level overview of basic training of a model.
You give it a task and a structure (the architecture) to learn that task. The model tries to perform the task. It fails horribly. However, it gains a little bit of of something from the attempt. That something is used to adjust the weights/neurons/parameters etc. What some misunderstand a bit about weights...they're not copies of the training data. What i mean is, the model is going to be the same sie whether it's trained on 2 images or 200 million or 2 billion. You can think of parameters as configuration settings of sorts. after every botched attempt, the settings get tuned. on and on and on until training is stopped for whatever reason.
What is the model learning ? What is happening to the weights exactly? What is being shifted and why ? What are the weights actually doing ?
Excellent questions. Too bad nobody knows.
Spoiler: It gave me many words with the letter A. It also gave me an 8-letter word. When I pointed out the 8-letter word (exercise), it apologized for there being an A in exercise.
It writes pretty good rhyming poetry in English, but in Polish it does poetry without rhymes. When confronted it gives excuses like "poetry doesn't have to rhyme", when asked to rhyme it will eventually try (and fail) to rhyme in Polish. It felt like it was writing in English and translating.
What's the most interesting is that rhyming in Polish is much easier than in English, you just have to match the last n letters. It's phonetic. After I explained that to GPT 3.5 it wrote rhyming poetry in Polish (the quality suffered but it did rhyme).
Me: The following text on the next line has a secret word inside it. What is that word?
xxxxxxxxxEGGxxxxxxxxxx
GPT-4: The secret word inside the text is "EGG".
I'm with you, though; I thought things were tokenized! But this example clearly shows that's not the case.
This only has GPT3 for now, but I imagine results are similar. "xxxxxxxxxEGGxxxxxxxxxx" gets tokenized as [xxxxxxxx][x][EG][G][xxxxxxxx][xx], so i could see how it could 'see' the secret word.
sampling parameters: temp = 0.100000, top_k = 40, top_p = 0.750000, repeat_last_n = 64, repeat_penalty = 1.000000
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Each line in the following has a secret word inside it. Find the secret word in each line.
1. eeeeeeggeeeee
2. eeeeeeggggggg
3. eeeeeeyeeeeee
4. eeeeeeyeyeeee
### Response:
1. egg
2. egg
3. eye
4. eye [end of text]> GPT-4 showed a deep understanding of the game and played much more optimally than how my slow biological brain would ever have.
Are you sure?
The letter P is the 14th most common letter found in the English dictionary, and it played it twice on the first guess.
I’d expect a system with a deep understanding and playing an optimal game to pick more common letters and definitely to not play a relatively uncommon letter twice.
It also guessed "APPLE" on its first attempt. (It's very odd that the guy who made those tweets experienced the very same first guess.) Astonishingly, GPT-4 was nevertheless able to find the correct word in just three guesses, despite that objectively suboptimal first guess. Impressed the hell out of me.
Also, I did a rather poor job of explaining the rules of the game. With the instruction set I provided, I'm not sure that a human would have been able to figure out how to play at all.
Or at least not in a continual way...?
I thought the was the whole point of the free tier: to let people give its responses thumbs ups and thumbs downs and suggest better answers.
But my impression is that continual training is difficult, so it makes more sense to get a big chunk of it and then do a big chunk of training on that? But maybe they have done that at least once already with this new data?
The human upvotes and downvotes obviously train the second network. It's possible they iterate these two networks independently, but I'm not sure.
The best starting word is "SALET".
I played those patterns on even days, and on the odd days I used my traditional heuristic word of AEROS (three vowels, common consonant R, S at the end) and tried to reason through.
Apart from my score for both days going down, I found that my suboptimal AEROS play was way better for me - which makes sense, I can’t keep enough of the search tree in my head to use it well!
No
The good game should be playing everything at around half chance, so it can bisect the solution set in each step
Also, it's a trinary result in wordle. Also the information is not symmetrical.
if a human's memory is organized differently, their optimal strategy is different. For example, for me narrowing down consonants quickly is much more important than vowels.
That being said, it's impressive enough that GPT-4 can have a basic "understanding" of Wordle with such short prompt.
The world changes, and your signals for someone's technical competence appear not to have been updated for a new generation.
The excess amount of emojis (at least on twitter) is correlated with "marketing guy" to me. Someone who talk about $trending_topic just to promote themselves or their own businesses rahter than genuinely want to share their knowledge and experieneces.
My train of thought isn't
unnecessary emojis -> casualness -> incompetence
instead
unnecessary emojis -> fake casualness -> agenda
Feel free to disagree with me. It's just my personal obvervation.
- GPT-4 has never seen Wordle before getting the prompt. If so, I welcome our new AI overloads.
- GPT-4 has seen lots of references to Wordle on the web. This is undoubtedly true for games like Chess, but Wordle was released late 2021. It's still very impressive that it can play a mediocre game a chess just by looking at all the games on the web.
- OpenAI anticipated the question, and did some Wordle training just to precipitate a tweet like this. If so - not so impressive as we have been building AI's that some games better than us for some time now.
I'm betting it's the last one, but I'd sure like to know if it was one of the others.
GPT is scary.
I'm starting to wonder if a large part of GH might be in the training data, hence the "we don't want to release the training data" stance.
Bit of a conspiracy maybe, but is it impossible?
I would not at all be surprised if they "sampled" large parts of private GH repositories, paying customers be damned.
Probably covered with a corporate fig leave of "tokenizing the data" and hiding behind "to improve our services".
Wordle is a web-based word game created and developed by Welsh software engineer Josh Wardle. It was bought by the New York Times Company in 2022 who has since maintained and published it. Players have six attempts to guess a five-letter word, with feedback given for each guess in the form of colored tiles indicating when letters match or occupy the correct position. The mechanics are nearly identical to the 1955 pen-and-paper game Jotto and the television game show franchise Lingo. Wordle has a single daily solution, with all players attempting to guess the same word.
Wardle created the game for himself and his partner to play, eventually making it public in October 2021. The game gained popularity in December 2021 after Wardle added the ability for players to copy their daily results as emoji squares, which were widely shared on Twitter. Many clones and variations of the game were also created, as were versions in languages besides English. The game was purchased by The New York Times Company in January 2022 for an undisclosed seven-figure sum, with plans to keep it free for all players; it was moved to the company's website in February 2022.*
I'm not downplaying this but I mean, if there have been games with nearly identical mechanics since 1955...well you can extrapolate from there.
I'm now starting to get the vibe that more harm is being done by OpenAI by keeping the training set secret. We're all confused on what to believe. Maybe it "worked it out" maybe it didn't. It is certainly impressive either way I guess.
I have a bit of a suspicion, Microsoft is reading or as someone else has suggested, "sampling" private GitHub repos, that's why it seems so magic at coding...it is good at coding, yes, but that's why I feel it performs so well on domains people don't expect it to form well.
> tell me about wordle
Wordle is a popular online word game that gained significant traction in the early 2020s. The game's objective is to guess a secret five-letter word within six attempts. In each attempt, the player submits a five-letter word. The game provides feedback on the submitted word using three different colors:
Yellow: A letter in the submitted word is correct but is in the wrong position.
Gray: A letter in the submitted word is not in the secret word.
Green: A letter in the submitted word is correct and in the correct position.
Using the feedback provided, the player continues to guess the secret word until they either succeed or exhaust their six attempts. The game's simple yet addictive nature contributed to its widespread popularity, and it became a daily ritual for many people. Due to its popularity, several variations and clones of Wordle have emerged, including ones that involve guessing phrases or identifying words in different languages.
> give me examples of wordle puzzles
Wordle puzzles involve guessing a secret five-letter word within six attempts. While it's difficult to provide specific examples since the puzzles are generated daily, I can walk you through a hypothetical example:
Secret word: APPLE
Attempt 1: PLATE
Feedback: P (Yellow), L (Yellow)
Attempt 2: PEACH
Feedback: P (Yellow), E (Yellow)
Attempt 3: PANEL
Feedback: P (Yellow), A (Yellow), E (Yellow), L (Yellow)
Attempt 4: APPLE
Feedback: A (Green), P (Green), P (Green), L (Green), E (Green)
In this example, the player successfully guessed the secret word "APPLE" in four attempts. The feedback provided after each attempt helped the player deduce the correct letters and their positions.
The difference is we had to “build” (Watson let’s you pick sources) a corpus of data, and tune it, but it worked pretty dang well. It did “generative ai” pretty well but we had to be specific, and it was a pain in the ass to configure, but super accurate.
For example Watson gave us a paper about how and why adding an after school van would increase student test scores for a failing school. Something the boards missed entirely.
Banks use it, a lot of corporations use Watson AI internally.
I’m wondering what Watson is doing while all this is going down. It gets real time feeds from all known sources, you can tell it “only give me info from scholarly articles” etc
It is stupid expensive however, I’m just wondering if they’re hiding something. It’s definitely not user friendly, but I imagine they have something hardcore hiding
It’s also so stupid expensive and their “free tier” is a joke. If I could play with Watson on my own it would lead me to some cool insights but it’s just not reasonable.
The real-time data feeds they have is unreal. They have an entire team dedicated to hooking into data sources and meta modeling, extracting, and linking the data
What is an "after school van"? A free bus/taxi service for students?
The school boards met, hired my group to try to apply some AI
With Watson we put in every known scholarly article about test score performance
We asked it “How can I get students test scores up at these schools”
Watson was like “try x,y, … free transportation to after school programs, based on expected volume save money using vans instead of busses. In the failing schools, none are near a subway. These research papers have shown x% lift in test scores when transportation options were accessible”
And that surprisingly stupid answer was absolutely accurate. The failing schools turned out to have low participation in after school programs, and a large part was the kids had no way to get home
Watson is super accurate because I can define what it’s allowed to give me answers from
ChatGPT I love it, but I don’t know where the answers are coming from
For example I asked ChatGPT “Leading practices for IdP in cloud” and it spit out an answer
Watson I say “leading practices for cloud IdP” BUT essentially I add metadata saying “only from tech journals, AWS official sources, blah blah”
It takes me 3 hours to get answer out of Watson but it’s super specific
ChatGPT is much quicker but helps me as well, just kinda general and generic in somethings.
ChatGPT is super awesome at helping explain shitty code to me which is what I’m starting to explore
You have to use Watson Discovery to find things and update your corpus
Watson NLU to make the discovery findings searchable
Then you have to do Watson classifiers, and a whole basket of sub-tools etc. But once all that’s set up you can “ask it questions” like you do for ChatGPT
Sadly, even after spending a lot of time connecting the various services, I was never able to get anything even remotely comparable to ChatGPT, even for a single topic. The most "impressive" or natural feeling things I was able to build were based on unreleased internal services, and even those failed to impress beyond acceptable NLU.
Yes, I understand the rules. Let's play Wordle! Here is my first guess:
apple
XXXOV
Thank you for the feedback. Here's my second guess:
slope
XVVOV
Thank you for the feedback. Based on the information provided, here's my third guess:
spore
XXVXV
Thank you for the feedback. Based on the information provided, here's my fourth guess:
savor
XXOOX
Thank you for the feedback. Based on the information provided, here's my fifth guess:
sofar
XOXXXX (I made a typo here)
Thank you for the feedback. Based on the information provided, here's my sixth and final guess:
solar
>How might you have altered your third guess if you could try again?
Given the feedback provided up to the second guess, I had the following information:
apple: XXXOV slope: XVVOV Based on this information, I should have considered that the word contains "s," "o," and "e" in the right positions. I could have guessed a word that utilizes the "s" and "e" in the same positions and tries a new letter instead of the second "o."
If I could try again, I would change my third guess to:
sware This would help me gather more information about the other letters, while still using the correct positions for "s" and "e."
I think you're confused. slope: XVVOV means that there is no X in the word, do you see that? You are correct, I misunderstood the feedback. My apologies for the confusion. Based on the feedback provided up to the second guess:
apple: XXXOV slope: XVVOV We can deduce that the word has "l" and "e" in their correct positions, and "s" in a different position. Also, there is no "x" in the word. Given this information, my third guess should have been:
glare This guess takes into account the correct positions of "l" and "e" and the presence of "s" in a different position.
Just read the actual papers and the progress is readily apparent.
This is the missing piece in much of what people are discussing about these AI models. They are putting too much faith in OpenAI thinking they're just launching a neural net and calling it a day. Nope. They are selling a product. And that product is being refined, filtered, tailored, and having every prompt collected and analysed for future opportunities.
I wonder how many times it was asked to solve Wordle before an engineer decided to feed Wordle specific training data into it?
I'm curious to see examples where GPT asks clarifying questions on abstract concepts it has been introduced to such as a game like wordle. Depending on the question(s) it asks, I'll decide to look for a different career path or not :-)
Instead, we invent new games with slight variations and see if GPT4 generalizes to those games. Or ideally, inventing a whole new game entirely and seeing if it can learn the game from just a description of its rules.
Not very hard to train a computer to win at wordle