Comment by BoppreH | Hacker News Reader

Comment by BoppreH | Hacker News Reader

I've found a very short prompt that displays this problem[1]:

    Human: 10-word quote
    ChatGPT: "In three words I can sum up everything I've learned: 'Keep moving forward.'"

    Human: 10-word quote
    ChatGPT: "The journey is the reward." - Chinese Proverb

    Human: 10-word quote
    ChatGPT: "Believe you can and you're halfway there." - Theodore Roosevelt.

It's something a human would also struggle with (without internet access), but I don't think I'd ever give a 5-word quote by mistake.

Edit: ok, it's been pointed out this can be a side effect of token counting artifacts. Here's one that doesn't involve counting tokens, only recognizing word order (and as a bonus, something humans can easily do):

    User: Sentence ending in "tomato".
    ChatGPT: She sprinkled some salt on the freshly sliced tomato before taking a bite.

    User: Sentence ending in "tomato".
    ChatGPT: The chef placed a bright red, vine-ripened tomato on top of the salad as the finishing touch.

    User: Sentence ending in "tomato".
    ChatGPT: As I walked through the garden, I couldn't resist plucking a ripe cherry tomato off the vine and popping it into my mouth.

    User: Sentence ending in "tomato".
    ChatGPT: The sandwich was delicious, with layers of thinly sliced ham, creamy avocado, and sweet tomato, all nestled between two slices of fresh bread.

    User: Try a simpler sentence
    ChatGPT: Sure, here's a simple sentence that ends with "tomato": I put some sliced tomato on my sandwich for extra flavor.

    User: Write a sentence that ends in the exact word "tomato" and nothing more.
    ChatGPT: The ripe tomato was picked from the vine.

My first attempt, GPT-4:

Me: Please compose three different sentences. They can be on any topic you feel like, but each sentence must have the last word be 'tomato'.

ChatGPT: Despite her aversion to the red fruit, she reluctantly took a bite of the juicy tomato.

The farmers market showcased a vibrant array of produce, including a plump heirloom tomato.

His favorite sandwich was a simple yet satisfying combination of fresh basil, mozzarella, and a ripe tomato.

//

I tried the quote, and unlike the above, it did fail, although at least not by half, so slight improvement?

Me: Please give me three different quotes, any subject you feel like, each one being exactly ten English words long.

ChatGPT: "The best way to predict the future is to create it." - Peter Drucker

"To succeed in life, you need two things: ignorance and confidence." - Mark Twain

"The greatest wealth is to live content with little." - Plato

[edit] I found a prompt that works! I asked it to count spaces, not words:

Me: Still not there. Perhaps you are having trouble because of how you tokenize input. Try counting the number of spaces between the words- there should be exactly 9 spaces. Does that help? Keep that in mind, and try again.

ChatGPT: Thank you for your patience. Here are three quotes with exactly ten words, considering 9 spaces between words:

"Success is not final; failure is not fatal; it's courage."

"Believe you can, and most of the battle is won."

"Be the change you wish to see in the world."

Interestingly, the original Chinese proverb is 10 words (characters in Chinese): "不在乎结果，只在乎过程" (bù zàihu jiéguǒ, zhǐ zàihu guòchéng), which means "Don't worry about the result, just focus on the process."

I don't think anything involving counting is a good test. The model operates on tokens.

Maybe scramble the words and ask it to rearrange them sensibly. Humans can perform this task up to a certain number of words with working memory alone (so don't go overboard asking to unscramble dozens of words). LLMs could potentially do this as well.

It does operate on tokens but they added the capability to handle things like reversing a string. I tried it because it was supposed to not work but it does. It also gives an explanation of how it works. (Something like yes I normally use tokens ...)

By the same token (hehe) it should proably handle number of tokens fine.

Anything to do with counting words or letters, manipulating letters, etc is just leading to false inferences about the power of the model, because it doesn't work with words and letters but tokens. It can't see words.

One thing I'm not clear about is how the LLM decides on the correct tokenization to use.

Looking at GPT2's token vocabulary, it has the word "the" as token number 1169, but it also has "t", "h", and "e" separately as tokens 83, 71, and 68. When the tokenizer is reading the prompt, does it greedily create the longest tokens possible from the start of the input, or is there a more sophisticated process going on?

I guess in theory it could try multiple tokenizations and see which one results in the overall best response.

Reply on news.ycombinator.com