This is a known design flaw of LLMs. This gets posted once a month for years. It’s disheartening to see very technically adept people still find this exciting. GPT3 has been out for years, why don’t more “hackers” know about tokenizers?

Basically, LLMs are “blind”. Fragments of text are converted to tokens, forming something like a big enum of possible tokens.

They can’t see spellings; they cannot see the letters.

So, they can’t handle things like “how many letters are in Mississippi?” reliably.

Due to chat bots running with nonzero temperature, they will sometimes emit a right answer just because the dice rolled in its favor. So if you go try this and get a good answer, that’s not conclusive either.

That’s the thing we’re dealing with, that’s how it works, that’s what it is.