Comment by marcodiego | Hacker News Reader

Comment by marcodiego | Hacker News Reader

I don't have access to ChatGPT4, but in my tests I could observe that it can't do some very simple tasks:

  - It can't play tic-tac-toe,
  - It can't play hangman,
  - It insists that winning on stone-paper-scissor using the chat (playing before me) is a matter of probability.

It was also demonstrated that it can't reverse strings.

Actually a transformer doesn't accesses 'strings', all it processes are tokens which are then mapped to vectors by whatever embedding is applied. I think it will be extremely difficult for a transformer to do any of these tasks correctly until a successor model is adopted.

I don't have much hope of any reasonably complex symbolic processing of anything that it was not trained on. Some of these tasks are easy for a human to perform with paper and pencil and a set of rules; of course a human may get confused, but for that you write programs. Write code is one of GPT's skills but It is not "that" good with code for problems that are not mere small modification of problems it was trained on.

EDIT: Could have expressed myself better: I don't have access to chatGPT4; I tested using the "available" chatGPT, I think it is 3.5.

A transcript of me trying to play tic-tac-toe with it: https://pastebin.com/V1CW5hpt

You're trying the old primary school method of testing things based on the wrong criteria. Why does it need to play tic-tac-toe, hangman, or rock paper scissors? Why do you think a language model would be good at those things?

Similarly, why would you expect a thing trained on the context of text would be good at reversing strings? It's amazing it's as good as it is at these things because it doesn't really make sense that it could do these things unless they trained it on reversed strings to add diversity but it's hard to gain context from a string backwards.

ALSO: your transcript doxs you and is hard to tell where your messages end and GPT begins. Just a heads up in case you don't want your full name leaked to whoever reads this

Because those games are just a way to measure how an internal state changes with moves initiated by the ego and initiated by someone else. The point is that there is no consistant internal state because it hallucinates and spotaneously changes. its like telling the language model a story, and getting it to repeat certain facts about it or you make additions to the story. its the exact same thing. It needs quite a lot of "prompt engineering" to push it in the correct direction, and even then its frequently incorrect.

It can play tic-tac toe, chess just fine

https://pastebin.com/cPwpZnZu

https://twitter.com/zswitten/status/1631107663500304384

The idea for games is that these are simple games, presumably with complete descriptions from various sources already in their database. The test is of their reasoning ability and ability to generalize descriptions into tasks that aren't chimeras of examples already in their database, y'know, like a classic test of what we'd call raw reasoning and intelligence, not crystallized intelligence.

With reversing strings unfortunately it's not likely to achieve that considering its tokenization.

You said you don't have access but based on your tests... Were you testing ChatGPT

I just tried and it was able to play tic tac toe, reverse a string (the string was "hello world.i am new to this so please forgive me if i can't reverse a sentence")

Hangman sort of worked but it said every letter I picked was correct and appears to have constructed a word based on my guesses. Very strange behavior

Try making it reverse this: "Quickly, the kangaroo hopped away, escaping under the azure sky."

I couldn't make it reverse that correctly even after prompting it five times to fix its mistakes.

Most commonly it writes: ".yks eruz a eht rednu gnipacse ,yawa depoh ooragnak eht ,ylkciuQ"

It also can't find the mistakes in there for the life of it.

Remember that GPT is working on input tokens and output tokens. Its output is tokens that then get converted back into text.

Taking [21063, 306, 11, 262, 479, 648, 38049, 45230, 1497, 11, 25071, 739, 262, 35560, 495, 6766, 13] to expect it to output back [13, 88, 591, 13724, 4496, 304, 4352, 2266, 28803, 19967, 541, 330, 325, 837, 88, 6909, 390, 381, 1219, 267, 273, 4660, 461, 304, 4352, 837, 2645, 74, 979, 84, 48] is a difficult problem that it is not well suited for.

That's because it's trained on the relations of words to each other and not on string manipulation. This is not its purpose. It may be capable of it to some degree but that seems like more of a luck of the draw kind of thing than something we should expect it to be good at.

It is insane that it can get that close, and it’s actually more impressive that it makes small typos than if it didn’t.

right? It makes it feel like it's actually trying to do it versus just having some backdoor'd string reverse function

In my experience GPT4 poorly performs ROT13 but can do base64 decoding really well. A lot of the early jailbreaks used base64 to sneak tokens into prompts. How could it base64 decode but not reverse a string? That's very odd.

My guess: Decoding Base64 is easy because it's a 1:1 mapping between strings. Since it's not supposed to be an encryption or obfuscation, there must be huge lookup tables somewhere on the internet that it uses as Rosetta stones.

The other thing is, it's trained on a lot of HTML, which includes data: URLs that decode to JS and SVG (on which it is also trained). So that one transformation is probably the one that is really baked well into the weights now.

BTW it doesn't just decode it, it also encodes it quite happily - in real time, as it is producing the output, sometimes unprompted. I once had GPT-4, when asked to produce SVG, produce it in form of an <img src="data:data:image/svg+xml;base64,..."> - and when I copy-pasted and rendered it, it was a valid SVG file with shapes inside.

How did you prompt it to play tic-tac-toe? I'm surprised that didn't work, it feels like something it should be able to handle really well.

Hangman and stone-paper-scissors though are entirely unsuited to a language model, at least one with a chat interface like ChatGPT, because they both require it to be able to store a secret. ChatGPT has no ability to do this: each time it returns a response by evaluating the previous conversation.

You could build a system that COULD play those games via an LLM but you'd have to write extra code to do it.

Well, for hangman at least, if the human knows the secret, it should be possible for the LLM to handle that, no?

Oh right - yeah, that would work great. I'd be very surprised if ChatGPT couldn't do that.

I asked Bing AI

> I am thinking of a five letter word. You guess the word letter-by-letter (in the style of the game hangman) as many times as you like, but may only guess incorrectly six times. Please begin guessing

> Sure! Let’s start with the first letter of the word you’re thinking of. What is it?

lol

but it did get the picture after that. It does seem to be very fixated on guessing vowels though, after it has already exhausted all of them.

It took a bit of prompt engineering but this worked for me in ChatGPT v4:

> Let's play hangman. I have thought of a 8 letter word, you have to guess it one guess at a time. You get to draw an ascii-art hangman too. Then YOU guess with a letter and I'll tell you if you were right or not. I won't give you a category clue. Start by drawing the hangman in a code block and then guessing a letter.

The same prompt against 3.5 didn't work - it didn't seem to be guessing likely next letters, and it couldn't keep track of how many body parts it should have drawn on the diagram.

On the free ChatGPT, I got it to play hangman with the prompt:

> Let's play hangman. I'm thinking of a word: _______

It made a variety of guesses generally in the form of

> Based on the pattern you provided, I will guess the letter "I". Is there an "I" in the word?

My response was either "No [letter]" or an updated state of the word: "_RE_IE_".

LLMs don't see individual characters, they see individual words.

Actually they see tokens, which are, usually, parts of words.

I was pretty disappointed when I tried some basic music theory questions. There's plenty of music theory information out there in text form, but it couldn't reliably tell me the tritone substitution of an F7 chord. I explained all the reasoning behind it to the point that it could parrot back the right answer, but then it made the same errors when I asked for the tritone substitution of an Eb7 chord. I wonder if that's improved with 4.

I also found it lacking when asking for chord variations and inversions on guitar (gpt-4)

Tic tac toe (on GPT-4) works with this

https://pastebin.com/cPwpZnZu

Reply on news.ycombinator.com