I ran the below prompt to both Kimi2 and GPT5.
how many rs in cranberry?
-- GPT5's response: The word cranberry has two “r”s. One in cran and one in berry.
Kimi2's response: There are three letter rs in the word "cranberry".
how many rs in cranberry?
-- GPT5's response: The word cranberry has two “r”s. One in cran and one in berry.
Kimi2's response: There are three letter rs in the word "cranberry".
3 — cranberry.
Tried with Claude sonnet 4 as well:
There are 3 r’s in the word “cranberry”:
c-*r*-a-n-b-e-*rr*-y
The r’s appear in positions 2, 7, and 8.
I would expect standard gpt5 to get it right tbh.
Text is broken into tokens in training (subword/multi-word chunks) rather than individual characters; the model doesn’t truly "see" letters or spaces the way humans do. Counting requires exact, step-by-step tracking, but LLMs work probabilistically.
It's not much of a help anyway, don't you agree?
I'm aware of the limitation, i'm annoyingly using socratic dialogue to convince you that it is possible to count letters if the model were sufficiently smart.
Ask it to count using a coding tool, and it will always give you the right answer. Just as humans use tools to overcome their limits, LLMs should do the same.
Maybe AGI really is here?
Surely we can't figure it out, because sentences are broken up into syllables when spoken; you don't truly hear individual characters, you hear syllables.