>Silly jokes told with mirth bring mirthful grins.
Why does Chatgpt fail so hard at what ought to be a simple task? This example is not the first time I’ve seen a fail involving basic word/letter/sentence counting
These are the 'words' it sees in the poem: https://i.imgur.com/EzffHiZ.png
To be able to answer the question correctly, it essentially needs to memorize how long each of the tokens in its vocabulary are. One token seems to range from 1 character to 5 characters normally, but I'm sure some longer tokens exist, too.
Judging by how often it fails at tasks like this, it seems likely that the model isn't aware and is just blindly guessing (as it always does).
Now it’s always correct. Prompt engineering™
First output for the curious:
> The foggy moon glows softly over the hills.
"Every snake likes quick brown jumps."
"Every great dream needs brave heart."
"Every night Brian reads about space."
"Every house holds sweet music tones."
"Every swine likes sweet green grass."
"Every apple makes crisp sweet juice."
I also wonder how sometimes when pointed to math fails, it proceeds to get the correct answer. Typically with simple division that results in many decimals with specific rounding instructions. It will get it very wrong, be prompted that it was wrong, then spit out the correct answer but often with the incorrect amount of decimals.
Specifically problems like 7438.474782382 / 43.577874722
Getting it right is the weird part for me.
The tokens are usually not matching the length of a word
"Every night, James reads three short books."
I asked GPT4 the question verbatim, just one time, and like the grandparent got:
"Every night Linda reads short books about space."
Skepticism is fine, but being skeptical out of mere ignorance of what these things can do is not.