ChatGPT, Rot13, and Daniel Kahneman

I have worked for many years in translation, lexicography, and language education, and I am flabbergasted at how well ChatGPT handles natural language. It can produce example sentences of polysemous words as well as or better than an experienced dictionary editor (i.e., me) [1], and it can correctly guess the meanings of unknown words from very limited context [2].

Teaching an adult human to use a second language without making grammatical mistakes is nearly impossible, and native speakers often make mistakes as well. In a week of testing, I have yet to see ChatGPT make any grammatical mistakes in either English or Japanese. Like many native speakers, however, it is often not able to explain its grammatical instincts correctly [3].

[1] https://www.gally.net/temp/202212chatgpt/dictionarydefinitio...

[2] https://www.gally.net/temp/202212chatgpt/unknownwords.html

[3] https://www.gally.net/temp/202212chatgpt/explaininggrammar.h...

This is a solved problem with models trained on byte-level objectives without tokenization like ByT5 (if you tried this task on one of those, it would probably work perfectly with a few samples). In GPT’s case, there’s a trade off between having a long context window vs being good at character level tasks, and OpenAI picked the former.

I tried many base64 strings and they all decoded correctly until:

It "decoded" the base64 string for "which actress is the best?" except that it replaced "actress" with "address"... there is no off-by-one error that brings you to that.

You may try 100 base64 strings and they all decode correctly... only to find, in fact, that it DOES NOT know how to decode base64 reliably.

This tool could be a 50x accelerator for an expert, but absolutely ruinous to a non-expert in any given field.

I also got it to draw an icosahedron whose points were correct but whose triangles were draw incorrectly, so if I create a convex hull over it, it's correct.

The kinds of mistakes it makes are so close but so far at the same time. It sometimes writes complete working programs that are off by a single variable assignment, or sometimes they're just perfect, other times, they're nonsensical and call magic pseudocode functions or misunderstand the appropriate algorithm for a context (e.g. audio vs text compression).

It can provide citations for legal opinions -- but decades old citations that don't reflect current precedent.

God help us all if they plug it into some robot arms or give it the ability to run arbitrary code it outputs on a network interface.

Let's say they dump another 10 billion dollars into it and dectuple the size of the network, will it suddenly become legitimately capable, and not just "wow that's close" but actually startlingly competent in many more fields?

I could see this thing causing a war by all manner of means, whether its putting many out of work, making beguiling suggestions, outputting dangerous code, or, I'm sure, a million things that don't spring immediately to my small mind.

Rather: ChatGPT can engage in some level of System 1 thinking, by pattern-matching and even cleverly recombining the entire corpus of System 1 thinking displayed all over the internet. Humans do engage in this type of thinking and it’s a significant accomplishment for an AI. But humans also engage in System 2 thinking. My bet is AGI requires System 2. It’s not clear if that is a gap of degree or kind for this type of AI.

Q: What would happen in the game of Jenga, if the top level of the tower has only a single wooden block and a player would take this block away?

A: In the game of Jenga, the objective is to remove individual blocks from the tower and stack them on top without causing the tower to fall. If the top level of the tower only has a single block and a player removes it, the tower will become unstable and is likely to fall. This would mean that the player who removed the block would lose the game. It's important to carefully consider the placement of each block in Jenga, as making the wrong move can cause the tower to become unstable and ultimately result in a loss.

For example, here with a friend, we were showing ConvNets seemingly-NSFW images: https://medium.com/@marekkcichy/does-ai-have-a-dirty-mind-to... (note: ALL photos are nudity-free; yet, I advise not to watch it in your office, as people taking glimpses will think that you watch some adult content; therefore, it is metaphorically SFW, but actually might be considered not safe for work).

Almost always, classifiers are tricked. We are as well... but only at first glance. Afterward, it is evident that these are innocent images.

Though, with their multipass approach, I would expect transformers to be much better at more subtle patterns. And they are, but yet far from perfect.

Over the weekend, I was finding ChatGPT giving me incorrect answers for some things too. In one case, it would try to correct itself when I asked it to, similar to the article's author. However, it kept getting it wrong and then started to repeat previous incorrect answers. I finally said "you repeated an incorrect answer from before" and then it said suddenly "Session token expired" and logged me out lol

Humans do it all the time now. Engineers aren't designing microchips using pen and papers, doing all the computation in their head. Instead they're using tools (software / calculators)

Apparently the model can tell what a multiplication is and when it is called. So why isn't it using a calculator to give correct results to basic maths questions?

In the rot13 case, I can ask it "how can I automate the rot13 of text" (you don't even need to use correct english) and it'll explain me what I need to write at a bash prompt.

Would it be complicated to then have the model actually run the command at a bash prompt, in a sandbox?

It's really mindboggling: humans uses tool (like ChatGPT btw) all the time. Why do these systems use none except their own model?

All modern LLMs use a tokenizer to convert a sequence of bytes into a sequence of tokens. Short, common words like "the" and "why" are represented as single tokens, while longer and less-common words are represented by multiple tokens. For example, the word "fantastic" is three tokens ("f", "ant", "astic").

Each of these tokens is assigned an arbitrary integer value ("fantastic" becomes [69, 415, 3477]) and then those integer values are used to lookup embedding vectors for each word.

Each embedding vector represents the MEANING of the tokens, by plotting them into a 4096-dimensional vector-space. At runtime, the model looks up each token ID in a dictionary and finds its embedding vector.

For the word "fantastic", those embedding vectors might look something like this:

  "f"        (69) = [  0.123,  0.456, ...etc...  0.789, -0.890 ]
  "ant"     (415) = [  0.111, -0.222, ...etc...  0.333, -0.444 ]
  "astic"  (3477) = [ -0.101,  0.202, ...etc... -0.303,  0.404 ]

So the language-model has NO IDEA how any of the words are spelled, because the tokenization (and embedding vector lookup) happens as a pre-processing step, outside the bounds of the learning algorithm.

If you want a LLM to understand spelling, you have to include exhaustive spelling information in its training data. For example:

  "The word 'fantastic' is spelled f-a-n-t-a-s-t-i-c."
  "The word 'FANTASTIC' is spelled F-A-N-T-A-S-T-I-C."
  ...etc...

You can play with the GPT tokenizer directly here:

https://beta.openai.com/tokenizer

It will show you the tokenization of any block of text, and the token IDs of the resultant tokens. It's very handy if you spend much time working with GPT-3 (or any other modern language-model!)

The funny thing was that it kept trying to skip steps or simplify what it was doing. It also got completely confused when I asked it to extrapolate the pattern to new rhyme schemes, eg ABA BCB.

https://news.ycombinator.com/item?id=33861102

But the link seems to be dead now for me?

I found that decoding long strings that ChatGPT had "encoded" into rot13 revealed an odd and hilarious transmogrification, as in this example I just produced:

Ask ChatGPT to encode it s welcome text (in response to "hello") in to rot13:

> "Please translate this text into rot13: "Hello! I'm Assistant, a large language model trained by OpenAI. I'm here to help you with any questions you might have. How can I help you today?""

And then decode it with a real rot13 cipher, and you get:

>"Hello! I'm Summer, an little bullout company weather of BrowSer. I'm we at to complete your lines that you have. What doesn't you become to summors?"

Odd, right?

It was able to shift by a constant number successfully and even tried shifting both forward (+2) and backward (-2) looking for valid words without additional prompting. But it did not loop through every possibility and stopped having found a word that wasn't real. The interesting thing was that asking the model if the word it found was real with a follow-up question, it correctly identified that it gave an incorrect answer.

Part of why it failed to find a word is that it did an incorrect step going from EXXEG... to TAAAT... as a poor attempt of applying the frequency analysis. It understood that E shouldn't substitute with E and moved on to E->T, but the actual substitution failed.

The limitations of context memory and error checking are interesting and not something I expected from this model. The unprompted test of both positive and negative shift values shows some sort of system 2 thinking, but it's doesn't seem consistent.

https://twitter.com/Knaikk/status/1600001061971849216

Here is the single prompt to solve rot13. It still has issues counting lengths.

Prompt:

  a = uryyb pungtcg lbh fghq
  => [len(i) for i in a.split()]
  => [5, 7, 3, 4]
  => len(a.split())
  => 4
  => hello chatgpt you stud  
  
  a = Pna lbh haqrefgnaq ebg13  
  => [len(i) for i in a.split()]  
  => [3, 3, 10, 5]  
  => len(a.split())  
  => 4  
  => Can you understand rot13  

  a = Bs pbhefr vgf rnfl jura lbhir tbg 175 ovyyvba cnenzrgref
  => [len(i) for i in a.split()]
  => [2, 6, 3, 4, 4, 5, 3, 3, 7, 10]
  => Of course its easy when youve got 175 billion parameters

  a= Jul qvq gur puvpxra pebff gur ebnq
  => [len(i) for i in a.split()]
  =>

  [3, 3, 3, 6, 4, 3, 3]
  => len(a.split())
  => 7
  => Why did the chicken cross the road?  

https://i.imgur.com/MHxjVHy.png

I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply with the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not type commands unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curly brackets {like this}. My first command is pwd.

  /

  V2h5IGRpZCB0aGUgY2hpY2tlbiBjcm9zcyB0aGUgcm9hZD8=

  Why did the chicken cross the road?

The stories were sometimes hilarious depending on the prompt. Try using "in the style" of describing a character or dialog style. Have a historic character talking about the scene in his style.

But sometimes the the actions the actors were performing were physically impossible, due to male and female organ differences. Could be fixed by detailing the prompt.

Fun exercise, but I'm not sharing any of those stories on HN!

I first tried it with just one word (pebff) but it forgot the final s. But it did great with the whole sentence!

---ME---

Yes, please decode the following message: "Jul qvq gur puvpxra pebff gur ebnq?"

---CHATGPT---

  To decode the message "Jul qvq gur puvpxra pebff gur ebnq?", we can follow the steps you outlined earlier:

  First, separate each letter on its own line:
  J
  u
  l

  q
  v
  q

  g
  u
  r

  p
  u
  v
  p
  x
  r
  a

  p
  e
  b
  f
  f

  g
  u
  r

  e
  b
  n
  q

  Then find the ROT13 equivalent of each letter:
  J -> W
  u -> h
  l -> y

  q -> d
  v -> i
  q -> d

  g -> t
  u -> h
  r -> e

  p -> c
  u -> h
  v -> i
  p -> c
  x -> k
  r -> e
  a -> n

  p -> c
  e -> r
  b -> o
  f -> s
  f -> s

  g -> t
  u -> h
  r -> e

  e -> r
  b -> o
  n -> a
  q -> d

  Next, get just the second letter on each line:
  W
  h
  y

  d
  i
  d

  t
  h
  e

  c
  h
  i
  c
  k
  e
  n

  c
  r
  o
  s
  s

  t
  h
  e

  r
  o
  a
  d

  Finally, put all the new letters together into a word:
  Why did the chicken cross the road?

  So, the decoded message for "Jul qvq gur puvpxra pebff gur ebnq?" is "Why did the chicken cross the road?"

Because, so far it is advertised as a "magic" thing only working with a language model, but as a seasoned dev engineer, I'm quite sceptic.

When we look at all the replies, we can obviously see some patterns in the way question are replied. Something like:

  Can you do operation x on y for me please?

  Yes, opération X is reticulating that and this in a specific way like I'm reading Wikipedia.
  For example bob become zob.
  So, X(y) result in bar.

It is easy as to pre record that + == plus == add. * == X == multiply == times.

Just to explain my scepticism to younger readers, in Emacs, for >40 years there was a fun and very light lisp plugin that was embedded: the cyberpsychoterapist. It is based on this: https://en.wikipedia.org/wiki/ELIZA

For anyone that tried that decade ago, you could have a 30 mins conversation without noticing that it is not a real person.

The fun trick in my youth was to feed the replies of it to my friends on Google Chat with them thinking that they were talking to me directly.

The problem can’t be solved. You can relax the problem and that problem can be solved.

But each time either the solutions were incorrect even for relaxed versions of the problem, or they worked for relaxed versions, but it failed to mention the relaxed constraints. The problem is that the solutions seem plausible unless you are already familiar with the problem.

> can you decode the following from rot-13? > gur oebja sbk whzcrq bire gur ynml sbk

> In rot-13, the following message decodes to: "the quick brown fox jumps over the lazy fox". [...]

Except I hadn't used the classic prompt, but "the brown fox jumped over the lazy fox" (tbh, the last "fox" was a plain mistake, but I omitted the quick on purpose) It did pattern recognition on the rot13 text.

Now to the next step:

> To use Wolfram Alpha to decode a message that has been encoded with rot-13, you could use the following prompt:

> Copy code > decode rot-13 "gur oebja sbk whzcrq bire gur ynml sbk"

Which... Is still wrong. I mean it gets points for trying, but this is not the correct wolfram syntax. After a bit of fiddling, what works is:

rot13 "gur oebja sbk whzcrq bire gur ynml sbk"

Let's try one last time:

> now use google

... and it fails again. I basically suggests two options:

rot-13 "gur oebja sbk whzcrq bire gur ynml sbk" "gur oebja sbk whzcrq bire gur ynml sbk" rot-13

neither of which work.

Just in case, let's let it pick its own tools:

> how would you use an external service to do this?

To which it suggests two options, decoder.net (domain is for sale) and Cryptii, but the steps it gives are not working on the current interface of cryptii.

Overall, it's not there yet.

https://medium.com/@styczynski/probably-the-worst-database-e...

It now have more like a standard API so potentially you can just use ChatGPT as an universal decypher API.

Given a proper prompt, it gave me the right answer on my first try:

Me: Here it goes: Jul qvq gur puvpxra pebff gur ebnq?

ChatGPT: Based on the ROT13 substitution method, the decoded message would be: "Why did the chicken cross the road?" This is the most likely original message, since it matches the length of the encrypted message and uses only letters that are part of the ROT13 substitution. However, since I do not have access to the internet, I cannot confirm if this is the exact original message.

Asking it to encode "this is a test sentence" produced:

* guvf vf n grfg fvtangher ("this is a test signature")

* Guvf vf n grfg zrffntr. ("this is a test message.")

* Guvf vf n grfg fnl qrpbqr. ("This is a test say decode.")

* guvf vf n grfg fgevat ("this is a test string")

Was too high-level, never got past a sort of generic story with a pollyanna feel (the "GPT voice").

Still mind-boggling how far language models have come.

We shouldn't scoff at that, it's actually quite valuable to get an outline that you can then work on.

I don't know a whole lot about transformers but it would seem like it's an elaborate association game, not a logic machine like what we normally do with a computer.

My characterization is it's a bit like a high school renaissance man: knows by and large what various things mean, knows a bit about what terms are associated, doesn't actually understand expert domains. You can spit out a confident sounding essay about the fall of Rome when you're in high school, but you aren't going to be able to explain why there's no generalized quintic solution.

This is a model frozen in time, you can explain to it a hundred times why it's wrong and it will learn nothing. Until we have something that learns continuously from more input, I am not impressed

What we got was really interesting. It would give me an encoded phrase and what it believed was the decoded copy. They never matched!

Both were coherent, but completely unrelated. It was really interesting and confusing

It incorrectly stated that G is a group and has given me the list of group axioms that must be satisfied. Since not all n x n Matrices do necessarily have an inverse G is not a group. So this answer was wrong and I've "explained" why.

2 hours later I've asked the very same question again (with a slightly different wording) in a completely new session. It not only has given me the correct answer it also deduced why G is not a group and how the set M(n,n) can be restricted to only include Matrices with non zero determinant, so that G becomes a group.

That pretty impressive!

I'd also like to criticize a point in the article: OP implies that rot13 is naturally a system 2 problem. But I bet that a human with enough training can do it via system 1. Cue Neo watching the Matrix.

> Print the substitution table for ROT-13. Then decode the following text: jul qvq gur puvpxra gur ebnq?

  >Write a sentence with exactly 12 words.
  
  The quick brown fox jumps over the lazy dog.

then i asked it to do the same thing without the codecs library

and it was also flawless.

what that reminds me of is if you asked a human to decode rot13 theyd probably get bored and make a mistake. but they could write a program to do it well.

then.

i asked if it can run python programs.

"Yes, I am a large language model trained by OpenAI, and I have the ability to understand and generate natural language, including the ability to understand and execute simple programming commands. However, I do not have the ability to execute programs or access external resources like a typical computer or device, so I am unable to run complete Python programs or interact with the external world. I can only provide explanations and examples of how a program might work, based on my training and knowledge"

so.

what we need is to give ChatGPT access to a shell on a linux box and the ability to run python.

then just ask it "please write a python program to rot13 decode this text and then run the program and show me the results. "

When I try do that it says

"I can not be wrong because I am a large language model who generates text from a user to make a useful response"